用音乐对齐的3D舞蹈生成框架,让动作更协调、情感更丰富。
Music-Aligned Holistic 3D Dance Generation via Hierarchical Motion Modeling
- 分层建模身体、手部和面部运动依赖关系
- 生成的舞蹈与音乐在时间与语义上高度同步
- 适合虚拟演出、游戏动画等需要真实舞蹈的应用
协调且与音乐对齐的全身3D舞蹈能显著增强情感表达和观众参与感。然而,由于缺乏全身3D舞蹈数据集、音乐与舞蹈跨模态对齐困难,以及身体各部位间运动耦合复杂,该任务仍具挑战。为此,我们构建了SoulDance——一个通过专业动捕系统采集的高精度音乐-舞蹈配对数据集,包含精细标注的全身舞蹈动作。基于此数据集,提出SoulNet框架,用于生成音乐对齐、运动协调的全身3D舞蹈序列。SoulNet包含三个核心组件:(1) 分层残差向量量化,建模身体、手部和面部间的复杂细粒度运动依赖;(2) 音乐对齐生成模型,将分层运动单元组合成富有表现力且协调的整体舞蹈;(3) 音乐-运动检索模块,一个预训练的跨模态模型,作为音乐-舞蹈对齐先验,在生成过程中确保输入音乐与生成舞蹈在时间与语义上的同步与一致。大量实验表明,SoulNet在生成高质量、音乐协调且精准对齐的全身3D舞蹈序列方面显著优于现有方法。
原文摘要 · Abstract (English)
Well-coordinated, music-aligned holistic dance enhances emotional expressiveness and audience engagement. However, generating such dances remains challenging due to the scarcity of holistic 3D dance datasets, the difficulty of achieving cross-modal alignment between music and dance, and the complexity of modeling interdependent motion across the body, hands, and face. To address these challenges, we introduce SoulDance, a high-precision music-dance paired dataset captured via professional motion capture systems, featuring meticulously annotated holistic dance movements. Building on this dataset, we propose SoulNet, a framework designed to generate music-aligned, kinematically coordinated holistic dance sequences. SoulNet consists of three principal components: (1) Hierarchical Residual Vector Quantization, which models complex, fine-grained motion dependencies across the body, hands, and face; (2) Music-Aligned Generative Model, which composes these hierarchical motion units into expressive and coordinated holistic dance; (3) Music-Motion Retrieval Module, a pre-trained cross-modal model that functions as a music-dance alignment prior, ensuring temporal synchronization and semantic coherence between generated dance and input music throughout the generation process. Extensive experiments demonstrate that SoulNet significantly surpasses existing approaches in generating high-quality, music-coordinated, and well-aligned holistic 3D dance sequences.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。