Multimodal Fusion of Voice and Gesture Data for UAV Control
Xiaojia Xiang, Qin Tan, Han Zhou, Dengqing Tang, Jun Lai
Abstract
To enable unmanned aerial vehicle (UAV) operators to efficiently and intuitively convey their commands to a swarm of UAVs, we propose the use of natural and human-centric input modalities, such as voices and gestures. This paper addresses the fusion of input modalities such as voice and gesture data, which are captured through a microphone and a Leap Motion controller, respectively, to control UAV swarms. The obtained experimental results are presented, and the achieved performance (accuracy) is analyzed. Finally, combined human factor ergonomics test with a questionnaire to verify the method’s validity.
Topics & Concepts
GestureComputer scienceModalitiesMicrophoneController (irrigation)Artificial intelligenceControl (management)Sensor fusionSwarm behaviourSpeech recognitionComputer visionHuman–computer interactionBiologySound pressureSociologyTelecommunicationsAgronomySocial scienceHand Gesture Recognition SystemsTactile and Sensory InteractionsIndoor and Outdoor Localization Technologies