用分层掩码建模生成音乐驱动的双人互动舞蹈,同步性更强。
DuetGen: Music Driven Two-Person Dance Generation via Hierarchical Masked Modeling
- 分两阶段:先将双人动作编码为分层离散标记,再由两个掩码Transformer生成
- 在基准数据集上实现最高水平的动作真实感与音乐-舞蹈对齐度
- 适合舞蹈生成、虚拟演出等需要双人协同的创意应用
我们提出DuetGen,一种从音乐生成交互式双人舞蹈的新框架。该任务的核心挑战在于双人舞蹈互动的复杂性,要求舞伴之间及与音乐保持同步。受运动合成进展启发,我们设计了两阶段方案:将双人动作编码为离散标记,并基于音乐生成这些标记。为有效捕捉复杂互动,我们将两名舞者的动作统一表示以学习必要运动标记,并在两阶段均采用粗到细的学习策略。第一阶段使用VQ-VAE,在粗时间粒度上分离高层语义特征,在细粒度上提取低层细节,生成两个不同抽象层级的离散标记序列。第二阶段,两个生成式掩码Transformer分别映射音乐信号到这些舞蹈标记:第一个生成高层语义标记,第二个在音乐和语义标记条件下生成低层标记。两个Transformer均通过预测序列中随机掩码标记进行训练,使推理时能通过逐步填充空序列生成运动标记。通过分层掩码建模与专用互动表示,DuetGen实现了跨多种风格的同步且交互式的双人舞蹈生成。在基准双人舞蹈数据集上的大量实验和用户研究证明,DuetGen在动作真实感、音乐-舞蹈对齐度和伙伴协调性方面达到当前最优表现。
原文摘要 · Abstract (English)
We present DuetGen, a novel framework for generating interactive two-person dances from music. The key challenge of this task lies in the inherent complexities of two-person dance interactions, where the partners need to synchronize both with each other and with the music. Inspired by the recent advances in motion synthesis, we propose a two-stage solution: encoding two-person motions into discrete tokens and then generating these tokens from music. To effectively capture intricate interactions, we represent both dancers' motions as a unified whole to learn the necessary motion tokens, and adopt a coarse-to-fine learning strategy in both the stages. Our first stage utilizes a VQ-VAE that hierarchically separates high-level semantic features at a coarse temporal resolution from low-level details at a finer resolution, producing two discrete token sequences at different abstraction levels. Subsequently, in the second stage, two generative masked transformers learn to map music signals to these dance tokens: the first producing high-level semantic tokens, and the second, conditioned on music and these semantic tokens, producing the low-level tokens. We train both transformers to learn to predict randomly masked tokens within the sequence, enabling them to iteratively generate motion tokens by filling an empty token sequence during inference. Through the hierarchical masked modeling and dedicated interaction representation, DuetGen achieves the generation of synchronized and interactive two-person dances across various genres. Extensive experiments and user studies on a benchmark duet dance dataset demonstrate state-of-the-art performance of DuetGen in motion realism, music-dance alignment, and partner coordination.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。