让视频生成更懂物体互动,用思维链推理运动轨迹。
C-Drag: Chain-of-Thought Driven Motion Controller for Video Generation
- 用思维链推理物体间动态交互,而非仅生成单一对象轨迹。
- 在新构建的VOI数据集上,运动控制精度显著提升。
- 适合需要精准物理交互的视频生成研究者与开发者。
基于轨迹的运动控制已成为可控视频生成中一种直观高效的方法。然而,现有方法通常仅生成受控对象的运动轨迹,忽略了受控对象与其周围环境之间的动态交互。为解决这一问题,我们提出一种基于思维链的运动控制器C-Drag,用于可控视频生成。C-Drag不直接生成某对象的运动,而是先进行物体感知,再根据给定的运动控制,推理不同物体间的动态交互。具体包括一个物体感知模块和一个基于思维链的运动推理模块。物体感知模块利用视觉语言模型捕获图像中各类物体的位置与类别信息。思维链运动推理模块以该信息为输入,分阶段推理各受影响物体的运动轨迹,并将其输入扩散模型进行视频合成。此外,我们构建了一个新的视频物体交互(VOI)数据集,包含三种典型交互类型,提供可用于准确评估性能的物体运动轨迹。实验结果表明,C-Drag在多个指标上表现优异,尤其在物体运动控制方面领先。代码、模型与基准数据将开源于https://github.com/WesLee88524/C-Drag-Official-Repo。
原文摘要 · Abstract (English)
Trajectory-based motion control has emerged as an intuitive and efficient approach for controllable video generation. However, the existing trajectory-based approaches are usually limited to only generating the motion trajectory of the controlled object and ignoring the dynamic interactions between the controlled object and its surroundings. To address this limitation, we propose a Chain-of-Thought-based motion controller for controllable video generation, named C-Drag. Instead of directly generating the motion of some objects, our C-Drag first performs object perception and then reasons the dynamic interactions between different objects according to the given motion control of the objects. Specifically, our method includes an object perception module and a Chain-of-Thought-based motion reasoning module. The object perception module employs visual language models to capture the position and category information of various objects within the image. The Chain-of-Thought-based motion reasoning module takes this information as input and conducts a stage-wise reasoning process to generate motion trajectories for each of the affected objects, which are subsequently fed to the diffusion model for video synthesis. Furthermore, we introduce a new video object interaction (VOI) dataset to evaluate the generation quality of motion controlled video generation methods. Our VOI dataset contains three typical types of interactions and provides the motion trajectories of objects that can be used for accurate performance evaluation. Experimental results show that C-Drag achieves promising performance across multiple metrics, excelling in object motion control. Our benchmark, codes, and models will be available at https://github.com/WesLee88524/C-Drag-Official-Repo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。