仅用RGB图像实现精准空间推理,靠相机信息主动引导视觉理解。
SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Models
- 用相机参数作为主动引导信号,动态融合视觉与空间信息。
- 在三个基准上刷新纪录,尤其在跨视角一致性任务中大幅领先。
- 适合需要真实空间感知的机器人、自动驾驶等应用研究者。
大型视觉语言模型(VLM)虽具备强大多模态理解能力,但在3D空间推理(如距离估计、尺寸比较、跨视角一致性)方面仍表现不足。现有3D感知方法依赖额外3D信息或通过浅层特征融合增强纯RGB VLM。本文提出SpaceMind,一种专为仅基于RGB输入的空间推理设计的多模态大语言模型。采用双编码器架构,融合VGGT(空间理解编码器)与InternViT(2D视觉编码器)。核心思想是将相机表示视为主动引导模态而非被动元数据。具体地,模型在语言模型前引入轻量级相机引导模态融合模块,对空间令牌施加相机条件偏置,分配反映几何重要性的查询无关权重,并利用相机嵌入门控融合表征。实验表明,SpaceMind在VSI-Bench、SQA3D和SPBench三个基准上均达到新最优性能,显著超越开放与专有系统,在VSI-Bench和SPBench上优势明显,且在SQA3D上达到当前最佳水平。结果证明,相机引导模态融合是一种有效且实用的归纳偏置,可赋予VLM真正空间化智能。代码与模型权重将公开以支持后续研究。
原文摘要 · Abstract (English)
Large vision-language models (VLMs) show strong multimodal understanding but still struggle with 3D spatial reasoning, such as distance estimation, size comparison, and cross-view consistency. Existing 3D-aware methods either depend on auxiliary 3D information or enhance RGB-only VLMs with geometry encoders through shallow feature fusion. We propose SpaceMind, a multimodal large language model explicitly designed for spatial reasoning solely from RGB inputs. The model adopts a dual-encoder architecture, integrating VGGT as a spatial understanding encoder and InternViT as a 2D visual encoder. The key idea is to treat the camera representation as an active guiding modality rather than passive metadata. Specifically, SpaceMind introduces a lightweight Camera-Guided Modality Fusion module before the language model to replace shallow fusion. It applies camera-conditioned biasing to spatial tokens, assigns query-independent weights reflecting their geometric importance, and uses the camera embedding to gate the fused representation. Empirically, SpaceMind establishes new state-of-the-art results on VSI-Bench, SQA3D and SPBench, surpassing both open and proprietary systems on VSI-Bench and SPBench by large margins and achieving state-of-the-art performance on SQA3D. These results demonstrate that camera-guided modality fusion is an effective and practical inductive bias for equipping VLMs with genuinely spatially grounded intelligence. We will release code and model checkpoints to support future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。