arXiv:2502.17414cs.CV2025-02ICCV被引 22

用一张图生成随音乐舞动的逼真长视频,无需额外训练。

X-Dancer: Expressive Music to Human Dance Video Generation

  • 用统一的变换器-扩散模型生成与音乐同步的2D姿态序列。
  • 在多个数据集上生成视频的多样性和真实感均超越现有方法。
  • 适合想快速生成舞蹈视频的研究者或创意工作者。

我们提出X-Dancer,一种新颖的零样本音乐驱动图像动画流程,可从单张静态图像生成多样且长时序的逼真人舞蹈视频。其核心是一个统一的Transformer-扩散框架,包含一个自回归Transformer模型,用于合成扩展且与音乐同步的2D身体、头部和手部姿态标记序列,这些序列随后指导扩散模型生成连贯真实的舞蹈视频帧。不同于主要在3D空间生成人体运动的传统方法,X-Dancer通过建模广泛的2D舞蹈动作,克服数据限制并提升可扩展性,利用易于获取的单目视频捕捉动作与音乐节拍的细微对齐。首先,我们从带关键点置信度的2D人体姿态标签构建空间组合标记表示,编码大范围肢体运动(如上下半身)与精细动作(如头部和手部)。接着设计音乐到动作的Transformer模型,自回归生成与音乐对齐的姿态标记序列,并结合全局注意力以捕获音乐风格和先前运动上下文。最后,利用扩散主干网络通过AdaIN将参考图像与合成姿态标记进行动画化,形成全可微分的端到端框架。实验表明,X-Dancer能生成兼具多样性和表现力的舞蹈视频,在多样性、表现力和真实感方面显著优于当前最优方法。代码与模型将用于研究目的开放。

原文摘要 · Abstract (English)

We present X-Dancer, a novel zero-shot music-driven image animation pipeline that creates diverse and long-range lifelike human dance videos from a single static image. As its core, we introduce a unified transformer-diffusion framework, featuring an autoregressive transformer model that synthesize extended and music-synchronized token sequences for 2D body, head and hands poses, which then guide a diffusion model to produce coherent and realistic dance video frames. Unlike traditional methods that primarily generate human motion in 3D, X-Dancer addresses data limitations and enhances scalability by modeling a wide spectrum of 2D dance motions, capturing their nuanced alignment with musical beats through readily available monocular videos. To achieve this, we first build a spatially compositional token representation from 2D human pose labels associated with keypoint confidences, encoding both large articulated body movements (e.g., upper and lower body) and fine-grained motions (e.g., head and hands). We then design a music-to-motion transformer model that autoregressively generates music-aligned dance pose token sequences, incorporating global attention to both musical style and prior motion context. Finally we leverage a diffusion backbone to animate the reference image with these synthesized pose tokens through AdaIN, forming a fully differentiable end-to-end framework. Experimental results demonstrate that X-Dancer is able to produce both diverse and characterized dance videos, substantially outperforming state-of-the-art methods in term of diversity, expressiveness and realism. Code and model will be available for research purposes.

舞蹈生成扩散模型零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。