Skeleton-based Hand Gesture Recognition using Geometric Features and Spatio-Temporal Deep Learning Approach
Abu Saleh Musa Miah, Jungpil Shin, Md. Al Mehedi Hasan, Yusuke Fujimoto, Nobuyoshi Asai
Abstract
Dynamic hand gesture recognition using a 3D skeleton dataset has become the most attractive research domain because of the multipurpose application. Although many researchers have been working to develop hand gesture systems, they are still facing challenges in achieving satisfactory performance and more generalizable properties because of the various complexities such as lacking effective features, computational complexity and slow execution speed etc. In the study, we proposed a selected joint skeleton feature selection approach along with CNN-based spatial and Multi-head attention network (MHAN) based temporal feature extraction to alleviate the problems. In the procedure, we selected the most effective skeleton point based on the visualized capability to extract geometric features and motion speed considering slow and faster motion speed. After enhancing the feature with CNN based spatial model, we produced a final joint skeleton-independent spatial feature vector. After that, we enhanced temporal contextual information by feeding them into MHAN; we applied a classification module to refine the feature with classification. We used two benchmark datasets to evaluate the model: DHGD and SHREC'17. The high performance of the proposed model proved the superiority of the proposed model.