Sign Language CV App.
Computer-vision-based sign language learning and gesture recognition application
Grade 11
ВугледарWhy Mykyta Telychko created the Sign Language CV App.
Right now, more than 430 million people around the world live with disabling hearing loss. For many of them, sign language is an essential part of communication, education, and everyday life. However, learning a sign language can be difficult when there are limited opportunities for regular practice, especially for people who are learning independently or live far from communities where sign language is widely used.The challenge is even greater when someone wants to learn a sign language that is not commonly taught in their local area. Access to qualified teachers, practice opportunities, and learning resources can vary greatly between countries and communities. This can make the first steps in learning a new sign language frustrating and difficult to maintain.Growing up in Vuhledar, a small city in the Donetsk region, I became interested in programming and technology from an early age. I wanted to use these skills to address a problem that affects millions of people and explore how technology could make sign language learning more accessible to anyone who wants to learn
How the Sign Language CV App works.
The app starts with a reference sign demonstrated through a video. The user selects a word they want to learn and watches how the corresponding sign is performed.When the user repeats the gesture in front of a camera, the computer vision system detects 21 key landmarks on the hands using MediaPipe. It tracks the position of these points in real time and uses their X and Y coordinates to describe the structure of the gesture.A mathematical algorithm then compares the distances between the detected landmarks with the reference gesture. This allows the system to evaluate how closely the user’s hand position matches the target sign and provide visual feedback on the result.The system was tested with 20 participants, including one participant with hearing impairments. Each participant completed 150 signs, resulting in 3,000 evaluated gestures. The average sign recognition accuracy reached 96.3%.
Are you a student inventor?
If you have an idea to solve a real problem in your community, we want to hear from you.
0 comments.
Sign in to join the conversation.