让视觉语言模型像人一样想象空间布局,提升复杂场景推理能力。
Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulators

- 用世界模拟器生成视角变化下的虚拟图像,辅助模型进行空间推断。
- 在MMSI-Bench上将推理准确率从29.8提升至38.8,最高达49.5。
- 适合需要空间推理的智能体、机器人导航与交互系统研究者。
尽管视觉语言模型(VLMs)具备较强的视觉推理能力,其空间推理仍受限于观察到的图像和文本链式思考。当仅提供有限的视角信息时,模型难以推断未观察到的布局,保持多视角一致性,或从不同视角进行推理。本文将此问题视为“以想象思考”,提出Astra框架,通过与世界模拟器交互,主动生成想象中的视觉证据。Astra由强化学习训练的Astra-VL策略与基于Bagel的世界模拟器Astra-WM组成,后者可根据上下文图像和自然语言描述的相机运动生成新视角观测。为确保想象内容可靠,Astra-WM采用视图一致性调优,提升视角间姿态与内容一致性。在强化学习阶段,设计了双阶段闭环课程,稳定工具使用探索,并促使模型仅在想象观测优于直接回答时才调用模拟器。实验表明,模拟器与代理策略均不可或缺:在MMSI-Bench上,Astra-WM使Gemini-3-Flash性能从45.1提升至49.5;Astra-VL使Qwen3-VL在MMSI-Bench上从29.8升至38.8,在MindCube上从36.8升至42.7。结果证明,想象观测可提供有效空间证据,但高效融合世界模型的推理需学习何时、何地、如何想象。
原文摘要 · Abstract (English)
While Vision-Language Models (VLMs) have shown strong visual reasoning capabilities, their spatial reasoning abilities remain largely constrained to the observed images and text-oriented chain-of-thought. They often struggle to infer unobserved layouts, maintain cross-view consistency, and reason from alternative viewpoints when only limited egocentric observations are available. In this work, we study this problem as thinking with imagination, where a VLM actively acquires imagined visual evidence by interacting with a world simulator during reasoning. We propose Astra, an agentic spatial reasoning framework that empowers VLMs with action-conditioned visual imagination. Specifically, Astra couples Astra-VL, an RL-trained VLM policy, with Astra-WM, a Bagel-based world simulator that generates novel-view observations from context images and natural-language camera motions. To provide reliable imagined evidence, Astra-WM is trained with view consistency tuning to improve pose and content consistency across views. In the RL stage, we propose a world-simulator-in-the-loop two-phase RL curriculum to stabilize tool-use exploration and advance the model's ability to invoke the simulator only when imagined observations improve over direct answering. Experiments demonstrate that both the world simulator and the agentic policy are necessary: Astra-WM improves simulator-augmented Gemini-3-Flash on MMSI-Bench from 45.1 to 49.5, while Astra-VL improves the Qwen3-VL backbone from 29.8 to 38.8 on MMSI-Bench and from 36.8 to 42.7 on MindCube. These results show that imagined observations can provide useful spatial evidence, but effective world-model-augmented reasoning requires learning when, where, and how to imagine.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。