Wearable Electronic Glove and Multilayer Para-LSTM-CNN-Based Method for Sign Language Recognition
Dapeng Wang, Mingyuan Wang, Ziqi Zhang, Teng Liu, Chuizhou Meng, Shijie Guo
Abstract
Communication between normal people and hearing-impaired people is usually realized with the sign language. However, the communication quality is limited by the familiarity of the people with sign language. Wearable sign language recognition systems, especially those based on flexible electronic glove, have attracted much attention due to their efficiency for sign language recognition. However, there are still two problems need to be solved in current research: 1) the size or weight of wearable device is usually inversely proportional to the fineness of the hand movement information captured and 2) both spatial features and temporal features are important for sign language recognition, and current algorithms usually do not achieve sufficient extraction. To address the above problems, a wearable electronic glove with new multilayer parallel LSTM-CNN (Para-LSTM-CNN) network is proposed. The designed wearable electronic glove is only 126 g in weight, and utilizes inertial measurement units (IMUs) and Flex sensors to capture hand motion signals. 41-dimensions of hand motion signals can be captured and uploaded to computer via Wi-Fi. A Para-LSTM-CNN algorithm based on parallel features extraction strategy is proposed. The temporal and spatial features of the sensor signals are extracted through long short-term memory (LSTM) and convolutional neural network (CNN), respectively. The Para-LSTM-CNN can effectively improve the accuracy of sign language recognition by fusing spatial and temporal features synchronously. The proposed system has a high sign-language-recognition accuracy of 97.01% for 26 gestures. Result shows that the proposed system promises to provide effective solutions for patients with hearing-impaired. The available code of Para-LSTM-CNN can be found at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/1104162390-A/Para-LSTM-CNN/tree/main</uri>.