用视频生成模型模拟物体交互的动态过程,支持可控输入。
InterDyn: Controllable Interactive Dynamics with Video Diffusion Models
- 基于视频扩散模型,输入初始帧和控制信号生成交互动态视频。
- 在未见物体上仍能生成合理且时间一致的交互视频。
- 适合需要动态交互模拟的研究者,如机器人、虚拟现实领域。
预测相互作用物体的动力学对人类和智能系统都至关重要。然而,现有方法局限于简化玩具场景,难以推广到复杂真实环境。生成模型虽可基于干预预测状态转移,但仅生成单一未来状态,忽略了交互带来的连续动态。为此,我们提出InterDyn框架,给定初始帧和编码驱动物体运动的控制信号,生成交互动态视频。核心思想是大型视频生成模型可作为神经渲染器与隐式物理“模拟器”,从大规模视频数据中学习交互动力学。为有效利用此能力,我们引入交互控制机制,使视频生成过程受驱动实体运动条件约束。定性结果显示,InterDyn能生成复杂物体交互的合理、时序一致视频,并泛化至未见物体。定量评估表明,其优于聚焦静态状态转移的基线方法。本工作揭示了利用视频生成模型作为隐式物理引擎的潜力。项目页面:https://interdyn.is.tue.mpg.de/
原文摘要 · Abstract (English)
Predicting the dynamics of interacting objects is essential for both humans and intelligent systems. However, existing approaches are limited to simplified, toy settings and lack generalizability to complex, real-world environments. Recent advances in generative models have enabled the prediction of state transitions based on interventions, but focus on generating a single future state which neglects the continuous dynamics resulting from the interaction. To address this gap, we propose InterDyn, a novel framework that generates videos of interactive dynamics given an initial frame and a control signal encoding the motion of a driving object or actor. Our key insight is that large video generation models can act as both neural renderers and implicit physics ``simulators'', having learned interactive dynamics from large-scale video data. To effectively harness this capability, we introduce an interactive control mechanism that conditions the video generation process on the motion of the driving entity. Qualitative results demonstrate that InterDyn generates plausible, temporally consistent videos of complex object interactions while generalizing to unseen objects. Quantitative evaluations show that InterDyn outperforms baselines that focus on static state transitions. This work highlights the potential of leveraging video generative models as implicit physics engines. Project page: https://interdyn.is.tue.mpg.de/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。