让视觉语言模型像人一样在3D空间中主动探索,提升空间推理能力。
Think3D: Thinking with Space for Spatial Reasoning
- 引入3D操作工具,让模型通过主动探索实现空间思维链。
- 在多个数据集上显著提升大模型的空间推理表现,小模型也能学会有效探索。
- 无需人工轨迹指导,仅靠最终答案奖励就能让小模型掌握3D空间策略。
尽管视觉语言模型(VLM)在2D视觉理解上表现优异,但其受限于2D范式,难以实现真正的3D空间推理。为此,我们提出Think3D框架,赋予VLM智能体交互式3D思维链推理能力。通过集成一系列3D操作工具,Think3D将感知转化为主动空间探索,模拟人类几何推理过程。该框架在BLINK Multi-view、MindCube-1K和VSI-Bench-Tiny等多个基准上持续提升GPT-4.1与Gemini 2.5 Pro等专有模型性能。我们进一步提出Think3D-RL,使小型开源模型如Qwen3-VL-4B在仅使用最终答案奖励、无过程监督或人工轨迹的情况下,自主学习有效的3D探索策略。训练后,模型展现出与强模型相似的工具使用模式,并将3D工具对MindCube-1K的性能影响从下降转为显著提升。结果表明,3D空间中的主动探索是一种高效且通用的空间推理改进范式。代码、模型与数据已公开于https://github.com/zhangzaibin/spagent。
原文摘要 · Abstract (English)
While Vision-Language Models (VLMs) excel at 2D visual understanding, they remain constrained by 2D-centric paradigm that severely limits genuine 3D spatial reasoning. To bridge this gap, we introduce Think3D, a novel framework that equips VLM agents with interactive, 3D chain-of-thought reasoning capabilities. By integrating a suite of 3D manipulation tools, Think3D transforms perception into active spatial exploration, mirroring human geometric reasoning. Think3D consistently improves proprietary models, including GPT-4.1 and Gemini 2.5 Pro, across BLINK Multi-view, MindCube-1K, and VSI-Bench-Tiny. We further propose Think3D-RL to teach smaller open-weight models how to manipulate 3D space effectively. Using only final-answer rewards, without process supervision or handcrafted exploration trajectories, Think3D-RL enables Qwen3-VL-4B to autonomously learn effective 3D exploration strategies. After training, the model exhibits tool-use patterns similar to those of stronger proprietary models, while shifting the effect of 3D tool use on MindCube-1K from a performance drop to a substantial improvement. These results show that active exploration in 3D space provides an effective and general paradigm for improving spatial reasoning in multimodal agents. Code, models, and data are available at https://github.com/zhangzaibin/spagent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。