Overview
This project addresses the communication gap between American Sign Language (ASL) users and non-signers by creating a robotic system that observes ASL gestures and transcribes them into physical written text. Utilizing a custom-trained YOLOv5 model, the system detects hand signs in real-time via webcam and converts them into text. A Pincher X 150 robotic arm then writes the recognized characters onto a whiteboard.
Technical Highlights
- Computer Vision: Implemented real-time ASL detection using YOLOv5, enhanced with stabilization algorithms to confirm consistent gesture detection.
- SVG Path Processing: Converted SVG font glyphs into optimized robot-executable paths, ensuring smooth and natural handwriting.
- Robot Control: Developed algorithms for dynamic roll adjustment and parabolic path mapping, overcoming the 4-DOF arm limitations for consistent pen contact.
Challenges and Solutions
- Robotic Limitations: Addressed constraints of a 4-DOF robotic arm through innovative kinematic solutions to maintain pen-to-surface contact.
- Gesture Recognition: Improved model robustness against varied lighting and user conditions through dataset augmentation and training optimization.
Technologies and Tools
- YOLOv5, PyTorch, OpenCV
- ROS Noetic, Python, Ubuntu 20.04
- Pincher X 150 robotic arm, Interbotix SDK
My Role
I contributed to developing the robotic control algorithms, specifically addressing dynamic orientation adjustments and optimized path planning for effective handwriting execution.
Key Results
The system successfully transcribed ASL signs into readable text on a whiteboard, demonstrating reliability, consistent legibility, and real-time responsiveness. Extensive testing confirmed the robustness of both the vision detection and robotic writing components.