用音乐或动作标签生成连贯的人体运动视频。
Video Motion Graphs
- 基于条件检索与插帧融合，生成匹配输入条件的视频。
- 在跳舞等复杂动作场景中仍能实现无缝插帧，保持轨迹准确。
- 适合需要多模态控制的人体动画生成任务。
我们提出 Video Motion Graphs 系统，用于生成逼真的人体运动视频。该系统以参考视频为基础，结合音乐或动作标签等条件信号，通过检索匹配条件的视频片段，并生成插值帧以无缝连接片段边界来合成新视频。核心是 HMInterp 模型，一种鲁棒的视频帧插值（VFI）方法，可处理复杂运动场景如舞蹈中的不连续帧。HMInterp 采用双分支插值策略：一是人体骨骼运动插值的运动扩散模型，二是基于扩散的视频帧生成模型。二是采用条件渐进训练，有效利用强条件（如图像）和弱条件（如姿态）。该设计确保高纹理质量和准确运动轨迹。实验表明，Video Motion Graphs 在多模态条件人体运动视频生成上优于现有生成式与检索式方法。项目页面见 https://h-liu1997.github.io/Video-Motion-Graphs/
原文摘要 · Abstract (English)
We present Video Motion Graphs, a system designed to generate realistic human motion videos. Using a reference video and conditional signals such as music or motion tags, the system synthesizes new videos by first retrieving video clips with gestures matching the conditions and then generating interpolation frames to seamlessly connect clip boundaries. The core of our approach is HMInterp, a robust Video Frame Interpolation (VFI) model that enables seamless interpolation of discontinuous frames, even for complex motion scenarios like dancing. HMInterp i) employs a dual-branch interpolation approach, combining a Motion Diffusion Model for human skeleton motion interpolation with a diffusion-based video frame interpolation model for final frame generation. ii) adopts condition progressive training to effectively leverage identity strong and weak conditions, such as images and pose. These designs ensure both high video texture quality and accurate motion trajectory. Results show that our Video Motion Graphs outperforms existing generative- and retrieval-based methods for multi-modal conditioned human motion video generation. Project page can be found at https://h-liu1997.github.io/Video-Motion-Graphs/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。