Research on interactive design of real-time gesture recognition based on multimodal Transformer algorithm
Yutong Zhang
Abstract
This paper proposes a multimodal adaptive weighting mechanism, which uses the self-attention mechanism of Transformer to efficiently fuse multiple modal data such as vision, depth sensor and inertial sensor. This mechanism can dynamically adjust the weight of each modality according to different environmental conditions (such as lighting, noise, etc.), thereby improving the recognition accuracy of the system in complex scenes. Secondly, this paper introduces a multi-scale attention mechanism to adapt to the time scale characteristics of different gestures, ensuring that the system can accurately capture the detailed changes of fast and slow gestures. In order to improve the real-time and adaptive capabilities of the system, this paper adopts an incremental learning strategy, which enables the model to automatically update parameters during real-time interaction and avoid retraining of the entire model. At the same time, through the adaptive optimization algorithm, the system can dynamically adjust the model processing strategy according to environmental changes, further enhancing the robustness in a changing environment. Experimental results show that the real-time gesture recognition algorithm based on multimodal Transformer proposed in this paper shows high accuracy and real-time performance under experiments, and has good application prospects.