Action Jitter Killer: Joint Noise Optimization Cascade for Skeleton-Based Action Recognition
Ruyi Liu, Yi Liu, Wentian Xin, Qiguang Miao, Long Li
Abstract
Skeleton-based action recognition is a crucial but challenging task in the application of engineering algorithms. However, due to the inaccurate estimation quality, certain joints that should theoretically lack dynamic information show irregular jitter, which affects the recognition accuracy. In this paper, we propose a Cascade Interaction Spatial-temporal transformer Network ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CI-STFormer</i> ) to address abnormal joint jitter and multi-scale aggregation of features effectively. The network comprises three parts: (1) Spatial sparse-level optimization and interaction module. Relying on the topology map, the sparse-level interaction adjacency matrix is constructed through cascading interaction to mask the jitter non-discriminative joints, and then the transformer is applied to realize the sparse-level internal feature update. (2) Spatial primal-level fusion and interaction module. Multi-head self-attention is employed to enhance the original feature representation. Then, the sparse-level update features are cascaded to construct the scale-cascaded interaction adjacency matrix to achieve the balance of noise caused by the jitter and the fusion of different scale features. (3) Temporal domain scale-level TCN combines the multi-scale filter and the temporal self-attention channel interaction algorithm to extract different temporal features and the interaction and update of global temporal information. The experimental results show that the proposed method performs excellently on the four datasets of NTU RGB+D 60, NTU RGB+D 120, UAV-Human, and NW-UCLA and achieves state-of-the-art performance on the transformer-based track. Related code will be available on https://github.com/Xdu-Liu/CI-STFormer.git.