无需训练即可分解视频运动类型，实现更自然的组合生成。
Training-free Motion Factorization for Compositional Video Generation
- 先规划后生成：通过运动图推理每帧物体形状位置变化
- 三类运动解耦合成：静止、刚体、非刚体运动各得其位
- 兼容主流扩散模型，适合需要精准控制运动的生成场景
组合视频生成旨在合成具有多样化外观和运动的多个实例。然而，现有方法主要关注语义绑定，忽略了提示中指定的多样运动类别。本文提出一种运动解耦框架，将复杂运动分解为三类：静止、刚体运动和非刚体运动。该框架遵循‘先规划后生成’范式：(1) 规划阶段，在运动图上推理运动规律，获取每个实例在各帧中的形状与位置变化，通过结构化表示消除用户提示的语义模糊；(2) 生成阶段，以解耦方式调节不同运动类别的合成。基于运动提示的引导分支分别稳定静止区域外观、保持刚体几何、规范局部非刚体形变。关键在于两个模块与模型无关，可无缝集成至各类扩散模型架构。大量实验证明，该框架在真实世界基准上实现了出色的运动合成性能。代码已公开于 https://github.com/ZixuanWang0525/MF-CVG。
原文摘要 · Abstract (English)
Compositional video generation aims to synthesize multiple instances with diverse appearance and motion. However, current approaches mainly focus on binding semantics, neglecting to understand diverse motion categories specified in prompts. In this paper, we propose a motion factorization framework that decomposes complex motion into three primary categories: motionlessness, rigid motion, and non-rigid motion. Specifically, our framework follows a planning before generation paradigm. (1) During planning, we reason about motion laws on the motion graph to obtain frame-wise changes in the shape and position of each instance. This alleviates semantic ambiguities in the user prompt by organizing it into a structured representation of instances and their interactions. (2) During generation, we modulate the synthesis of distinct motion categories in a disentangled manner. Conditioned on the motion cues, guidance branches stabilize appearance in motionless regions, preserve rigid-body geometry, and regularize local non-rigid deformations. Crucially, our two modules are model-agnostic, which can be seamlessly incorporated into various diffusion model architectures. Extensive experiments demonstrate that our framework achieves impressive performance in motion synthesis on real-world benchmarks. Code is available at https://github.com/ZixuanWang0525/MF-CVG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。