让视觉语言模型学会提前规划多步视角变化,提升3D空间推理能力。
Planning with the Views

- 通过自探索与视图图蒸馏的迭代框架,构建视角间的连接关系。
- Qwen2.5-VL-7B在交互式视图规划任务上准确率从2.5%提升至47.8%。
- 适合研究多模态决策、3D场景理解与自主智能体的开发者参考。
能否让视觉语言模型(VLM)预测每一步相机移动对视角的影响,并提前规划多步动作?我们称之为视图规划,需具备(1)理解单步动作如何改变视角,(2)在多轮计划中组合多个变换以达成目标视角。我们在真实ScanNet场景的3D点云环境ViewSuite中测试了13个前沿VLM。结果发现:尽管模型具备基础视角-动作知识,但在多步组合中表现严重不足,且随视角距离增加差距扩大。为此,我们提出一种迭代框架,交替进行自探索与视图图蒸馏。核心思想是:所有探索轨迹(无论成败)共同构成一个视图图,紧凑地捕捉场景中视角间的连接关系。将该图蒸馏为多样化的监督任务,重塑策略分布,克服纯强化学习中的稀疏奖励问题。该方法使Qwen2.5-VL-7B在交互式视图规划任务上的准确率从2.5%提升至47.8%,超过GPT-5.4 Pro(18.5%)和Gemini 3.1 Pro(21.4%)。自探索成为推动VLM实现3D空间主动推理的重要路径。代码与数据见https://viewsuite.github.io。
原文摘要 · Abstract (English)
Can VLMs predict how each camera move changes the view, and plan many such moves ahead? We call this capability view planning, requiring (1)understanding how a single action transforms the view, and (2)composing many such transformations across multi-turn plans to identify a target view. We probe both abilities in our proposed ViewSuite, a 3D point-cloud environment on real ScanNet scenes. Across 13 frontier VLMs, a critical planning gap emerges: they possess basic view-action knowledge but fail to compose it across multi-turn plans, with the gap widening as viewpoint distance grows. To close this gap, we propose an iterative framework that alternates self-exploration with view graph distillation. The key insight is that all exploration trajectories, regardless of their outcome, collectively form a view graph that compactly captures how viewpoints connect across a scene. Distilling this graph into diverse supervised tasks reshapes the policy distribution and overcomes the sparse rewards that stall pure RL. This improves Qwen2.5-VL-7B from 2.5% to 47.8% on interactive view planning, surpassing GPT-5.4 Pro (18.5%) and Gemini 3.1 Pro (21.4%). Self-exploration emerges as a promising path toward VLMs that can actively reason and plan in 3D space. Code and Data are at https://viewsuite.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。