Sign Language to Text Translation with Computer Vision: Bridging the Communication Gap
So Xue Thong, Eng Lip Tan, Ching Pang Goh
Abstract
This research paper addresses the communication barriers faced by individuals using sign language and those without hearing difficulties. With limited adoption of sign language and a scarcity of proficient human translators, there is a need for innovative solutions. This research presents a real-time sign language translation system using computer vision technology. The system utilizes Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks for static and dynamic sign recognition, respectively. Word segmentation which utilized ‘word-ninja’ and a Large Language Model (LLM) contributed to accurate sentence generation. The proposed system integrates machine translation and text-to-speech functionalities to improve the system's accessibility. The methodology involves data collection, landmark recognition, and the implementation of recognition and translation models. The results show impressive accuracy which are 99.20% and 90.08% for static and dynamic sign recognition respectively. However, issues such as environmental conditions have affected the detection accuracy and made errors in recognising similar signs. Sentence generation also provided a pretty decent result which is 90% of the accuracy from the output provided by the LLM model. Despite these challenges, the system still contributes to reducing communication barriers and promoting inclusivity in society.