用舞蹈节奏模式生成动物同步跳舞的视频，只需6个关键帧
How Animals Dance (When You're Not Looking)
- 用节奏模式控制舞蹈结构，实现长时序动作协调
- 仅需6个关键帧即可生成30秒跨动物、跨音乐的舞蹈视频
- 适合对创意动画或生物运动模拟感兴趣的开发者
我们提出一个生成音乐同步、编舞感知动物舞蹈视频的框架。该框架引入编舞模式——定义舞蹈长期结构的运动节拍序列——作为舞蹈生成的新高层控制信号。这些模式可从人类舞蹈视频中自动估计。以少数关键帧（由文本生成图像或GPT-4o生成）为起点，将舞蹈合成建模为图优化问题，寻找满足指定节拍编舞模式的最佳关键帧结构。我们还提出一种镜像姿态图像生成方法，用于捕捉舞蹈中的对称性。中间帧通过视频扩散模型生成。仅需6个输入关键帧，即可在多种动物和音乐曲目上生成长达30秒的舞蹈视频。
原文摘要 · Abstract (English)
We present a framework for generating music-synchronized, choreography aware animal dance videos. Our framework introduces choreography patterns -- structured sequences of motion beats that define the long-range structure of a dance -- as a novel high-level control signal for dance video generation. These patterns can be automatically estimated from human dance videos. Starting from a few keyframes representing distinct animal poses, generated via text-to-image prompting or GPT-4o, we formulate dance synthesis as a graph optimization problem that seeks the optimal keyframe structure to satisfy a specified choreography pattern of beats. We also introduce an approach for mirrored pose image generation, essential for capturing symmetry in dance. In-between frames are synthesized using an video diffusion model. With as few as six input keyframes, our method can produce up to 30 seconds dance videos across a wide range of animals and music tracks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。