让AI像人一样用3D几何想象推理,仅凭有限视角就理解空间关系。
Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views
- 利用视觉语言模型生成3D潜在表示,实现无3D先验的动态空间想象。
- 在多个基准上超越强基线,推理准确率显著提升。
- 适合需要3D空间理解的机器人、自动驾驶等场景应用。
尽管视觉-语言模型在多模态任务中取得显著进展,但从有限视角理解3D空间关系仍是重大挑战。以往方法通常依赖纯文本(如拓扑认知地图)或2D视觉线索,其表征能力有限,难以应对需3D空间想象的任务。为此,我们提出3DThinker框架,能在推理过程中有效利用图像中的丰富几何信息,如同人类般进行3D心智模拟。该框架是首个无需3D先验输入即可实现3D心智模拟的方法,且不依赖显式标注的3D数据训练。训练分两阶段:首先通过监督学习对齐视觉语言模型生成的3D潜在表示与3D基础模型(如VGGT)的输出;其次仅基于结果信号优化整个推理轨迹,从而精炼底层3D心智模拟过程。跨多个基准的大量实验表明,3DThinker持续优于强基线,为将3D表征融入多模态推理提供了新思路。代码已公开于https://github.com/zhangquanchen/3DThinker。
原文摘要 · Abstract (English)
Though recent advances in vision-language models (VLMs) have achieved remarkable progress across a wide range of multimodal tasks, understanding 3D spatial relationships from limited views remains a significant challenge. Previous reasoning methods typically rely on pure text (e.g., topological cognitive maps) or on 2D visual cues. However, their limited representational capacity hinders performance in specific tasks that require 3D spatial imagination. To address this limitation, we propose 3DThinker, a framework that can effectively exploits the rich geometric information embedded within images while reasoning, like humans do. Our framework is the first to enable 3D mentaling during reasoning without any 3D prior input, and it does not rely on explicitly labeled 3D data for training. Specifically, our training consists of two stages. First, we perform supervised training to align the 3D latent generated by VLM while reasoning with that of a 3D foundation model (e.g., VGGT). Then, we optimize the entire reasoning trajectory solely based on outcome signals, thereby refining the underlying 3D mentaling. Extensive experiments across multiple benchmarks show that 3DThinker consistently outperforms strong baselines and offers a new perspective toward unifying 3D representations into multimodal reasoning. Our code is available at https://github.com/zhangquanchen/3DThinker.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。