arXiv:2605.24321cs.CV2026-05被引 2

一个模型搞定3D场景理解与交互,无需微调即可零样本完成多种任务。

Unified 3D Scene Understanding Through Physical World Modeling

论文配图:Unified 3D Scene Understanding Through Physical World Modeling
图 1 · 摘自论文原文
  • 用概率图模型统一表示多模态场景元素,通过不同推理路径实现任务切换。
  • 在新视角合成与物体操作上达到顶尖性能,无需微调且保持几何一致性。
  • 支持复杂组合任务,适合需要通用视觉理解的智能系统研发者。

3D场景理解需要灵活组合深度估计、新视角合成和物体操作等视觉推理任务,这些任务对感知与交互至关重要。现有方法通常孤立处理各项任务,难以共享表示或迁移知识。本文提出物理世界建模框架3WM(3WM),将其构建成概率图模型,节点代表RGB、光流、相机位姿等多模态场景元素。不同任务通过图中不同推理路径自然涌现:从RGB与密集光流提示生成新视角,从RGB与稀疏光流提示实现物体操作,从RGB与相机条件推断深度,全部零样本完成,无需任务专属训练。3WM在新视角合成与3D物体操作任务上超越专用基线模型,无需微调即可实现高精度控制、强几何一致性及真实场景鲁棒性。此外,模型支持可组合推理路径,如移动物体以导航环境,实现复杂几何推理。这表明统一模型可作为碎片化任务系统的实用替代,迈向通用视觉世界模型的重要一步。

原文摘要 · Abstract (English)

Understanding 3D scenes requires flexible combinations of visual reasoning tasks, including depth estimation, novel view synthesis, and object manipulation, all of which are essential for perception and interaction. Existing approaches have typically addressed these tasks in isolation, preventing them from sharing a common representation or transferring knowledge across tasks. A conceptually simpler but practically non-trivial alternative is to unify these diverse tasks into a single model, reducing different tasks from separate training objectives to merely different prompts and allowing for joint training across all datasets. In this work, we present a physical world model for unified 3D understanding and interaction (3WM), formulated as a probabilistic graphical model in which nodes represent multimodal scene elements such as RGB, optical flow, and camera pose. Diverse tasks emerge from different inference pathways through the graph: novel view synthesis from RGB and dense flow prompts, object manipulation from RGB and sparse flow prompts, and depth estimation from RGB and camera conditioning, all zero-shot without task-specific training. 3WM outperforms specialized baselines without the need for finetuning by offering precise controllability, strong geometric consistency, and robustness in real-world scenarios, achieving state-of-the-art performance on NVS and 3D object manipulation. Beyond predefined tasks, the model supports composable inference pathways, such as moving objects aside while navigating a 3D environment, enabling complex geometric reasoning. This demonstrates that a unified model can serve as a practical alternative to fragmented task-specific systems, taking a step towards a general-purpose visual world model.

3D理解统一建模零样本物理模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。