IDF-Sign: Addressing Inconsistent Depth Features for Dynamic Sign Word Recognition
Sunusi Bala Abdullahi, Kosin Chamnongthai
Abstract
Inconsistent hand and body features pose barriers to sign language recognition and translation leading to unsatisfactory models. Existing recognition models are built up on the spatial-temporal depth <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S<sub>p</sub></i> features. Finding suitable expert features for the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S<sub>p</sub></i> model is challenging especially for dynamic sign words because many inconsistent features exist across hand motions and shapes. In this article, we propose IDF-Sign: an efficient and consistent <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S<sub>p</sub></i> model from a spatial-temporal multivariate pairwise consistency feature ranking (PairCFR) approach. The temporal features are obtained by computing the 3D position vector of skeletal hand joint coordinates, while the spatial features were obtained by taking every ten spatial coordinates in the 3D video frames and averaging it and doing so until the end of the frames. The PairCFR was used to rank and select the best <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S<sub>p</sub></i> model features at different feature thresholds. We employed a threshold selection to compute a mid-point value of each ranked feature according to its weight. The receiver operating characteristics (ROC) scheme was employed to identify the relationship between the sensitive parameters and the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S<sub>p</sub></i> features, and the obtained values were utilized as modeling inputs. To verify the IDF-Sign, we design a real-life experiment with a leap motion sensor (LMS) consisting of ten signers with a total of ninety dynamic sign words. LMS provides the depth videos, since depth videos are too dense for the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S<sub>p</sub></i> model to treat directly, we read the depth videos in comma-separated files in real time. Extensive IDF-Sign evaluations using machine learning on ASL, GSL, DSG, and ASL-similar datasets prove the Optimized Forest achieved an average recognition performance of 95%, 78%, 65.07%, and 95.33% of the top-1, respectively.