让大模型逐步理解空间几何,提升对3D场景的推理能力。
Let Geometry GUIDE: Layer-wise Unrolling of Geometric Priors in Multimodal LLMs

- 分层注入几何特征,让语言模型逐步吸收空间信息。
- 在VSI-Bench上5B和9B模型分别达71.5和72.1分。
- 适合需要空间理解的多模态任务,如3D场景解析。
多模态大语言模型在2D视觉任务中表现优异,但在理解真实视觉流中的物理空间方面仍存在困难。近期基于前馈几何基础模型的方法虽能隐式提取几何先验,但多数模型仅从单一层级的编码器提取特征,并在语言模型输入端进行一次性融合,可能遗漏跨层级的互补信息,限制细粒度几何线索的获取,阻碍渐进式跨模态适应。为此,我们提出GUIDE(Geometric Unrolling Inside MLLM Early-layers),一种渐进式集成几何先验的框架。GUIDE从几何编码器中采样并对齐多层级特征,逐层注入到多模态大模型的早期层中。该设计使模型能持续获取多粒度几何线索,并在层间语义抽象过程中逐步融合视觉表征。GUIDE还引入双层次上下文感知门控机制,在令牌与层两个层面选择性调控几何信息,减少冗余注入与对预训练表征的干扰。在空间推理与3D场景理解基准测试(包括VSI-Bench、ScanRefer和Scan2Cap)上的大量实验验证了GUIDE的有效性与跨任务适用性。我们的5B和9B模型在VSI-Bench上平均得分分别为71.5和72.1。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have achieved remarkable progress in 2D visual tasks but still struggle to understand physical space in real-world visual streams. Recently, feed-forward geometric foundation models that implicitly extract geometric priors from visual inputs have offered a promising direction. However, many existing geometry-aware MLLMs extract features from a single deep encoder layer and perform one-shot fusion at the language-model input, potentially overlooking complementary information across geometric encoder layers, limiting access to fine-grained geometric cues, and hindering progressive cross-modal adaptation. To address these limitations, we propose GUIDE (Geometric Unrolling Inside MLLM Early-layers), a progressive framework for integrating geometric priors. GUIDE samples and aligns multi-level features from the geometric encoder and sequentially injects them into the early layers of the MLLM. This design enables the MLLM to continuously access multi-granularity geometric cues and progressively integrate them with visual representations during layer-wise semantic abstraction. GUIDE further introduces a dual context-aware gating mechanism at both the token and layer levels, selectively regulating geometric information to reduce redundant injection and interference with pretrained representations. Extensive experiments on spatial reasoning and 3D scene understanding benchmarks, including VSI-Bench, ScanRefer, and Scan2Cap, validate the effectiveness and cross-task applicability of GUIDE. Our 5B and 9B models achieve average scores of 71.5 and 72.1 on VSI-Bench, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。