Continuous Hand Gesture Recognition for Human-Robot Collaborative Assembly

Bogdan Kwolek

202312 citationsDOI

Abstract

In this work, we present a framework for dynamic hand gesture recognition on RGB images acquired by an overhead camera. The recognition is realized for Methods Time Measurement-based planning of human-robot collaborative workspace. The 3D hand posture is estimated by MediaPipe. The recognition is done by a neural network in which a layer-wise feature combination takes place. We combine features extracted by basic blocks of Spatio-Temporal Adaptive Graph Convolutional Neural Network and by basic spatio-temporal self-attention blocks. We recorded and manually annotated 12 videos consisting of 54,659 RGB images with five basic motion sequences: grasp, move, position, release, and reach. We demonstrate experimentally that results of our networks are superior to results achieved by RNNs, ST-GCN, ST-AGCN, and CTR-GCN networks.

Topics & Concepts

Computer scienceArtificial intelligenceRGB color modelConvolutional neural networkGesture recognitionComputer visionFeature (linguistics)Artificial neural networkRobotGraphGestureGRASPWorkspacePattern recognition (psychology)Human–robot interactionTheoretical computer scienceProgramming languageLinguisticsPhilosophyHand Gesture Recognition SystemsHuman Pose and Action RecognitionRobot Manipulation and Learning