DuetGen: Music Driven Two-Person Dance Generation via Hierarchical Masked Modeling

Abstract

Fig. 1.DuetGen generates synchronized two-person dance choreography from input music, featuring natural and close interactions between dancers.We present DuetGen, a novel framework for generating interactive twoperson dances from music.The key challenge of this task lies in the inherent complexities of two-person dance interactions, where the partners need to synchronize both with each other and with the music.Inspired by the recent advances in motion synthesis, we propose a two-stage solution: encoding two-person motions into discrete tokens and then generating these tokens from music.To effectively capture intricate interactions, we represent both dancers' motions as a unified whole to learn the necessary motion tokens, and adopt a coarse-to-fine learning strategy in both the stages.Our first stage utilizes a VQ-VAE that hierarchically separates high-level semantic features at a coarse temporal resolution from low-level details at a finer resolution, producing two discrete token sequences at different abstraction levels.Subsequently, in the second stage, two generative masked transformers learn to map music signals to these dance tokens: the first producing high-level semantic tokens, and the second, conditioned on music and these semantic tokens, producing the low-level tokens.We train both transformers to learn to predict randomly masked tokens within the sequence, enabling

Topics & Concepts