通过注入几何先验提升视觉语言模型的三维空间推理能力。
Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning

- 在Transformer层中注入几何先验,以学习基础空间规律而非依赖特定数据集。
- 对应匹配准确率提升至70%以上,时间鲁棒性超过85%,远超基线。
- 无需3D VQA数据训练,显著提升多个空间推理基准表现。
视觉语言模型在三维空间推理上表现不佳。现有方法依赖3D视觉问答数据微调,易受数据偏差影响;而引入专用3D视觉编码器则不够灵活。本文提出GASP(几何感知空间先验)框架,将基本几何先验直接注入大语言模型的Transformer层中。该方法通过一个小的对应头,在所有层施加深度监督信号,采用双目标训练:基于大规模视频场景的真实点对应关系的对比损失,强制实现2D视角不变性;深度一致性监督则解决3D几何歧义。分析表明,标准VLM内部对应匹配准确率常低于5%。经训练后,对应匹配峰值达70%以上,且时间鲁棒性保持在85%以上,而基线始终低于5%。这些内部改进带来下游任务显著提升:在All-Angles Bench上提升18.2%,在VSI-Bench上提升29.0%,且全程未使用任何3D VQA数据。结果表明,从基础几何先验学习是实现更可靠三维空间推理的可行且可泛化路径。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) often struggle with robust 3D spatial reasoning. Prevailing methods that rely on fine-tuning with 3D visual question-answering (VQA) datasets may overfit dataset-specific biases, while integrating specialized 3D visual encoders is often inflexible and cumbersome. In this paper, we argue that genuine spatial understanding should emerge from learning fundamental geometric priors, not only from high-level VQA supervision. We propose GASP (Geometric-Aware Spatial Priors), a framework that injects these priors directly into the LLM's transformer layers. GASP employs a small correspondence head, applied as a deep supervision signal across all layers, and is trained with a dual objective leveraging ground-truth geometry from large-scale video scenes: a contrastive loss on ground-truth point correspondences enforces 2D view-invariance, while a depth consistency supervision resolves 3D geometric ambiguities. Our analysis first provides a diagnostic showing that standard VLMs' internal correspondence matching accuracy is very low (often below 5%). We then demonstrate that our training substantially improves this behavior, boosting peak layer-wise correspondence to over 70% and maintaining over 85% temporal robustness while baselines remain below 5%. These internal improvements translate to significant gains on downstream spatial benchmarks including +18.2% on All-Angles Bench and +29.0% on VSI-Bench, all without training on any 3D VQA data. Our findings indicate that learning from fundamental geometric priors is a promising and generalizable pathway towards VLMs with more reliable 3D spatial reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。