用文字生成复杂人体动作视频,先转骨骼再合成画面。
Controllable Complex Human Motion Video Generation via Text-to-Skeleton Cascades
- 分两阶段:先由文本生成2D骨骼序列,再用骨骼驱动视频生成。
- 在自建2000视频数据集上,动作多样性与流畅性均领先。
- 适合需要精细控制复杂动作的动画、游戏开发者使用。
当前视频扩散模型难以生成翻滚、后空翻、武术等复杂人体动作视频。纯文本条件存在时间模糊问题,而基于姿态的控制需完整骨架序列,制作成本高。本文提出两级级联框架:首先通过自回归文本到骨骼模型,基于先前生成的姿态逐关节预测,捕捉复杂动作所需的长时序依赖与关节协同;其次,采用姿态条件视频扩散模型,结合参考图像与生成骨架序列合成视频,引入DINO-ALF(自适应层融合)多层级参考编码器,在大幅姿态变化和自遮挡下仍能保持外观与衣物细节。为弥补公开数据集不足,构建基于Blender的合成数据集,含2000个视频,涵盖多样角色与特技动作,支持外观、动作与环境自由控制。实验表明,该方法在自建数据集与Motion-X Fitness基准上,于FID、R-precision及动作多样性指标上优于现有方法;视频生成部分在VBench评估中,于时间一致性、动作平滑性与主体保留方面表现最佳。
原文摘要 · Abstract (English)
Generating videos of complex human motions such as flips, cartwheels, and martial arts remains challenging for current video diffusion models. Text-only conditioning is temporally ambiguous for fine-grained motion control, while explicit pose-based controls, though effective, require users to provide complete skeleton sequences that are costly to produce for long and dynamic actions. We propose a two-stage cascaded framework that addresses both limitations. First, an autoregressive text-to-skeleton model generates 2D pose sequences from natural language descriptions by predicting each joint conditioned on previously generated poses. This design captures long-range temporal dependencies and inter-joint coordination required for complex motions. Second, a pose-conditioned video diffusion model synthesizes videos from a reference image and the generated skeleton sequence. It employs DINO-ALF (Adaptive Layer Fusion), a multi-level reference encoder that preserves appearance and clothing details under large pose changes and self-occlusions. To address the lack of publicly available datasets for complex human motion video generation, we introduce a Blender-based synthetic dataset containing 2,000 videos with diverse characters performing acrobatic and stunt-like motions. The dataset provides full control over appearance, motion, and environment. It fills an important gap because existing benchmarks significantly under-represent acrobatic motions while web-collected datasets raise copyright and privacy concerns. Experiments on our synthetic dataset and the Motion-X Fitness benchmark show that our text-to-skeleton model outperforms prior methods on FID, R-precision, and motion diversity. Our pose-to-video model also achieves the best results among all compared methods on VBench metrics for temporal consistency, motion smoothness, and subject preservation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。