从视频中学习可变形物体的物理动态,实现高保真交互模拟。
DeformMaster: An Interactive Physics-Neural World Model for Deformable Objects from Videos

- 基于视觉观测推断物理状态,结合神经残差补偿未建模误差。
- 支持真实动作重演、材料参数调整与动态新视角合成。
- 适用于机器人交互、虚拟现实等需要物理感知的场景。
可变形物体的世界模型需同时捕捉几何、外观、物理动力学、交互基础及材料行为。从真实视频中学习此类模型极具挑战,因可变形线状、面状和体状物体在高维形变、噪声交互和复杂材料响应下演化。模型必须从视觉观测中推断物理状态,在新交互下向前滚动,并以高视觉保真度渲染动态变化。我们提出 DeformMaster,一种从视频中构建的交互式物理-神经世界模型,将真实交互视频转化为统一动力学与外观框架下的可交互可变形物体模型。DeformMaster 在保持结构化物理滚动的同时,使用神经残差补偿未建模效应;将稀疏手部运动建模为分布式的柔顺执行器以实现手-连续体交互;通过空间异质本构专家表示材料响应;并从预测的物理演化驱动高保真4D外观。在真实可变形物体序列上的实验表明,DeformMaster 能准确预测未来动态并渲染动态外观,优于现有最先进方法,且支持新型动作滚动、材料参数变化与动态新视角合成。
原文摘要 · Abstract (English)
World models for deformable objects should recover not only geometry and appearance, but also underlying physical dynamics, interaction grounding, and material behavior. Learning such a model from real videos is challenging because deformable linear, planar, and volumetric objects evolve under high-dimensional deformation, noisy interactions, and complex material response. The model must therefore infer a physical state from visual observations, roll it forward under new interactions, and render the resulting dynamics with high visual fidelity. We present DeformMaster, a video-derived interactive physics-neural world model that turns real interaction videos into an online interactive model of deformable objects within a unified dynamics-and-appearance framework. DeformMaster preserves structured physical rollout while using a neural residual to compensate for unmodeled effects, grounds sparse hand motion as distributed compliant actuator for hand-continuum interaction, represents material response with spatially varying constitutive experts, and drives high-fidelity 4D appearance from the predicted physical evolution. Experiments on real-world deformable-object sequences demonstrate DeformMaster's ability to roll out future dynamics and render dynamic appearance, outperforming state-of-the-art baselines while supporting novel action rollout, material-parameter variation, and dynamic novel-view synthesis. Project page: https://can-lee.github.io/deformmaster-web/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。