Hand-by-Hand Mentor: An AR based Training System for Piano Performance
Ruoxi Guo, Jiahao Cui, Wanru Zhao, Shuai Li, Aimin Hao
20212021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)21 citationsDOI
Abstract
Multimedia instrument training has gained great momentum benefiting from augmented and/or virtual reality (AR/VR) technologies. We present an AR-based individual training system for piano performance that uses only MIDI data as input. Based on fingerings decided by a pre-trained Hidden Markov Model (HMM), the system employs musical prior knowledge to generate natural-looking 3D animation of hand motion automatically. The generated virtual hand demonstrations are rendered in head-mounted displays and registered with a piano roll. Two user studies conducted by us show that the system requires relatively less cognitive load and may increase learning efficiency and quality.
Topics & Concepts
PianoMIDIComputer scienceHidden Markov modelAnimationMotion captureHuman–computer interactionAugmented realityTraining (meteorology)Speech recognitionQuality (philosophy)MultimediaMotion (physics)Artificial intelligenceComputer graphics (images)ArtArt historyEpistemologyPhilosophyOperating systemPhysicsMeteorologyMusic Technology and Sound StudiesHuman Motion and AnimationMusic and Audio Processing