Computer Vision ASL Detection Robotic Arm

This project creates a robotic hand that bridges communication between hearing and deaf communities. The system both recognizes American Sign Language (ASL) letters through computer vision and physically demonstrates them through mechanical movement.

The software stack consists of three main components: (1) Computer vision pipeline using OpenCV for hand tracking and feature extraction, (2) TensorFlow-based neural network trained on ASL gesture datasets for real-time classification, and (3) Arduino firmware for smooth servo control and interpolation.

The ML model was trained on 10,000+ labeled hand gesture images, achieving 95% accuracy on the test set. The system processes frames at 30 FPS, providing responsive gesture recognition.

• Raspberry Pi 3B+ or 4 (2 GB RAM minimum)
• PCA9685 servo driver board (16-channel, I2C)
• 5x micro servo motors (SG90 or MG90S)
• 30V/10A power supply for servos
• Pi Camera module or USB webcam
• 3D-printed hand components (STL files in GitHub repo)
• Fishing line (~50cm per finger for tendon actuation)
• Jumper wires, screws, and mounting hardware

The robotic hand requires several 3D-printed parts that work together to enable realistic finger movements. Download the STL files from the project repository and print the five finger components along with the palm and wrist structural pieces.

Print settings: - Material: PLA+ for strength and print quality - Layer height: 0.2mm - Infill: 30% for structural rigidity - Supports: Yes, for overhangs on finger joints

Each finger includes articulated joints designed in SolidWorks, with modular link sections that can be reprinted independently if damaged.

Construction involves threading fishing line through printed components to create finger actuation:

1. Cut approximately 50cm lengths of fishing line — one per finger.
2. Thread each line through the finger starting at the fingertip, routing through the joint channels.
3. Secure the line at the fingertip with a knot or crimp.
4. Route each line down through the palm to connect to its corresponding servo motor horn.
5. Ensure all joints move freely with minimal friction before tightening.

The design emphasizes lightweight construction while maintaining structural rigidity. Key design considerations include tendon routing for finger actuation, servo mounting points, and wire management.

The electrical system uses a PCA9685 servo driver board to control multiple servos via I2C communication with the Raspberry Pi:

1. Connect the PCA9685 to the Raspberry Pi: SDA → GPIO 2, SCL → GPIO 3, VCC → 3.3V, GND → GND.
2. Connect the 30V/10A power supply to the PCA9685 servo power terminals (V+ and GND).
3. Plug each servo into channels 0–4 on the PCA9685.
4. Connect your camera module to the Pi (CSI port for Pi Camera, or USB).

The architecture separates computationally intensive ML inference from real-time servo control, with the Pi handling both through serial communication.

Install Raspberry Pi OS via the official imager tool, then set up the software environment:

1. Enable SSH and I2C via raspi-config.
2. Update packages: `sudo apt update && sudo apt upgrade`
3. Install Python dependencies:

`pip install opencv-python mediapipe tensorflow adafruit-circuitpython-pca9685`

1. Clone the project repository:

`git clone https://github.com/ammarjmahmood/ASLRoboticHand.git`

Deployment was streamlined using Docker containers to ensure consistent environments. The system was successfully tested on Raspberry Pi 4, Raspberry Pi 5, MacBook Pro M2, and Windows PC.

The system uses MediaPipe to track hand landmarks, then trains a Random Forest Classifier to recognize letter patterns from these tracking points:

1. Run the data collection script to capture hand landmark data for each ASL letter.
2. Position your hand in front of the camera and cycle through letters A–Z.
3. The script extracts 21 hand landmarks (63 features: x, y, z per landmark).
4. Train the classifier on your collected dataset.
5. Test recognition accuracy — aim for 95%+ on your test set.

The system processes frames at 30 FPS, providing responsive gesture recognition with an average response time of 200ms from detection to physical replication.

Each ASL letter requires specific finger positions. Calibration ensures precise movements:

1. Run the calibration script to test each servo's range of motion.
2. For each letter, define the servo angles that produce the correct finger positions.
3. Fine-tune values — each servo and mechanical assembly has individual variations.
4. Save your calibration profile.

The completed system successfully recognizes and replicates 15 common ASL gestures with high accuracy.

Verify the complete system:

• All fingers move smoothly through their full range
• Fishing line shows no binding or excessive slack
• Servo horns are securely attached
• Camera has a clear view for hand detection

Advanced: Hand Mirroring Mode — A real-time mimicry mode allows the robotic hand to copy your hand movements live, rather than recognizing specific ASL letters. Run the hand_mirror program for this functionality.

Future improvements include expanding the gesture vocabulary, implementing bilateral hand design, and developing a standalone embedded system without requiring a computer for ML inference.