arXiv:2507.19836cs.GRcs.AI2025-07中稿 · the 33rd ACM Inter…被引 8

让舞蹈视频精准跟上音乐节奏并自定义风格,支持任意舞者和分辨率。

ChoreoMuse: Robust Music-to-Dance Video Generation with Style Transfer and Beat-Adherent Motion

  • 用SMPL参数做音乐与视频的中间桥梁,突破视频分辨率限制。
  • 新设计的MotionTune音频编码器使动作严格贴合节拍和音乐情绪。
  • 支持个性化舞者风格迁移,适合创意制作与艺术自动化场景。

现代艺术创作日益需要能适应多样音乐风格与个体舞者特征的自动编舞生成。现有方法常无法生成既贴合音乐节奏又符合用户指定舞蹈风格的高质量舞视频,限制了其在真实创作场景中的应用。为此,我们提出ChoreoMuse,一个基于扩散模型的框架,采用SMPL格式参数及其变体作为音乐与视频生成之间的中介,克服了传统视频分辨率的限制。ChoreoMuse支持风格可控、高保真度的舞蹈视频生成,适用于多种音乐类型与个体舞者特征,且可处理任意参考舞者在任意分辨率下的生成。方法引入新型音乐编码器MotionTune,从音频中捕捉运动线索,确保生成编舞紧密追随输入音乐的节拍与表现力。为定量评估生成舞蹈与音乐及编舞风格的契合度,我们提出两项新指标。大量实验表明,ChoreoMuse在视频质量、节拍对齐、舞蹈多样性与风格一致性等多个维度均达到领先水平,展现出作为广泛创意应用可靠解决方案的潜力。视频结果见项目页:https://choreomuse.github.io。

原文摘要 · Abstract (English)

Modern artistic productions increasingly demand automated choreography generation that adapts to diverse musical styles and individual dancer characteristics. Existing approaches often fail to produce high-quality dance videos that harmonize with both musical rhythm and user-defined choreography styles, limiting their applicability in real-world creative contexts. To address this gap, we introduce ChoreoMuse, a diffusion-based framework that uses SMPL format parameters and their variation version as intermediaries between music and video generation, thereby overcoming the usual constraints imposed by video resolution. Critically, ChoreoMuse supports style-controllable, high-fidelity dance video generation across diverse musical genres and individual dancer characteristics, including the flexibility to handle any reference individual at any resolution. Our method employs a novel music encoder MotionTune to capture motion cues from audio, ensuring that the generated choreography closely follows the beat and expressive qualities of the input music. To quantitatively evaluate how well the generated dances match both musical and choreographic styles, we introduce two new metrics that measure alignment with the intended stylistic cues. Extensive experiments confirm that ChoreoMuse achieves state-of-the-art performance across multiple dimensions, including video quality, beat alignment, dance diversity, and style adherence, demonstrating its potential as a robust solution for a wide range of creative applications. Video results can be found on our project page: https://choreomuse.github.io.

舞蹈生成音乐驱动风格迁移扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。