用无配对数据提升舞蹈音乐生成,让动作与音乐更同步。
Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment

- 结合预训练模型与对比学习,对齐舞步和音乐特征空间。
- 在AIST++数据集上实现更优的动作-音乐对齐效果。
- 适合舞蹈创作、自动伴奏等需要精准时序同步的场景。
舞蹈生成音乐是一项有前景的任务,可用于编舞辅助和自动伴奏,其中身体运动与声音的时间协调至关重要。使用人体关节位置作为运动表示具有优势:能显式捕捉身体动态,且轻量、保护隐私,易于集成到动捕和姿态估计流程中。然而,高质量的舞步-音乐配对数据稀缺,因准确同步的采集成本高,且受版权和表演权限制。这使得仅依赖配对数据训练端到端模型困难。为此,我们提出一种舞蹈条件音乐生成框架,高效利用无配对和配对数据。方法结合预训练的单模态编码器(运动与音乐)、基于节拍的对比预训练以对齐特征空间,并在预训练文本到音频扩散模型上添加类ControlNet的条件模块。在AIST++上的实验表明,该方法提升了舞蹈-音乐对齐性和音频质量,量化与定性评估均验证了其有效性。相比现有最优方法,本方案在舞蹈对齐性能上更优,音频质量具有竞争力。代码已公开于 https://github.com/kmraven/AudioLDM-ControlNet。
原文摘要 · Abstract (English)
Dance-to-music generation is a promising task for applications such as choreography support and automatic accompaniment, where temporal coordination between body movement and sound is essential. In particular, using human joint positions as the motion representation is attractive because they explicitly capture body dynamics while being lightweight, privacy-preserving, and easy to integrate with motion capture and pose-estimation pipelines. A central challenge in this setting, however, is the scarcity of high-quality paired dance-music data, since collecting accurately synchronized pairs is costly and often constrained by copyright and performance rights. This makes it difficult to train end-to-end models solely from paired data. To address this issue, we propose a dance-conditioned music generation framework that efficiently exploits both unpaired and paired data. Our method combines pretrained unimodal encoders for motion and music, beat-guided contrastive pretraining to align their feature spaces, and a ControlNet-style conditioning module on top of a pretrained text-to-audio diffusion model. Experiments on AIST++ demonstrate that the proposed techniques improve both dance-music alignment and audio quality, as confirmed by quantitative and qualitative evaluations. Compared to a state-of-the-art method, our approach achieves superior dance alignment performance and competitive audio quality. Code is available at https://github.com/kmraven/AudioLDM-ControlNet .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。