从单张图生成可交互的物理世界模型,支持长期探索与操作。
TourPhysics: Bringing Physics to World Models for Exploration and Manipulation from a Single Image

- 结合确定性物理模拟与视频生成,分离状态、几何、控制与外观记忆。
- 在长时程重访中显著减少外观漂移,保持场景一致性。
- 适合需要物理可信交互的虚拟环境构建与机器人训练。
交互式视觉世界模型需区分观测与物理干预:相机运动揭示新表面,而干预改变物体运动、接触与形变。当前视频世界模型多受外观先验驱动,长期预测中常丧失物理或空间一致性。我们提出TourPhysics,一种从单张图像和声明式物理配置初始化的在线框架。该框架扩展了我们2026年ACM Multimedia的工作PhysOmni,将有限物理驱动的视频合成推进至持久探索与操作。TourPhysics结合确定性模拟与视频生成,分别管理模拟器状态、几何证据、生成器控制与外观记忆。每一步动作中,模拟器先计算有限的物理与相机轨迹,再生成对应观测。被接受的观测发布终端状态,并更新外观记忆与后续生成控制,而模拟器状态与几何保持不变。我们进一步将投影可见性使用的模拟器几何与生成器条件用的相对深度分离开。参考锚定的记忆通过几何跨视角对应检索静态外观,以有界残差形式融合,无有效对应时回归原始路径。在模拟器定义的相机巡游与物体操作下,TourPhysics比基线方法更准确跟踪预定轨迹,保持输入场景,并减少长时程重访中的外观漂移。
原文摘要 · Abstract (English)
Interactive visual world models must distinguish observation from physical intervention. Camera motion reveals new surfaces, whereas intervention changes object motion, contact, and deformation. Current video world models are largely driven by appearance priors and often lose physical or spatial consistency over long horizons. We present TourPhysics, an online framework initialized from a single image and a declarative physical configuration. TourPhysics extends PhysOmni, our ACM Multimedia 2026 work, from finite physics-grounded video synthesis to persistent exploration and manipulation. TourPhysics combines deterministic simulation with video generation while assigning separate roles to simulator state, geometric evidence, generator controls, and appearance memory. For each action, the simulator computes a finite physical and camera trajectory before the corresponding observation is generated. Accepted observations publish the terminal state and update the appearance memory and subsequent generator controls, while the committed state and simulator geometry remain fixed throughout synthesis and retry. We further separate the simulator geometry used for projection and visibility from the relative depth used to condition the generator. A reference-anchored memory retrieves accepted static appearance through geometric cross-view correspondence and incorporates it through a bounded residual that reverts to the native path when no valid correspondence exists. On simulator-defined camera tours and object manipulations, TourPhysics follows prescribed camera and object trajectories more closely than the evaluated baselines, preserves the input scene, and reduces appearance drift during long-horizon revisits.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。