arXiv:2603.18795cs.CVcs.AI2026-03被引 1

让视觉语言模型显式生成空间信息,提升细节定位能力。

Perceptio: Perception Enhanced Vision Language Models via Spatial Token Generation

  • 用自回归序列直接生成语义分割和深度图标记
  • 在多个基准上显著提升空间理解准确率,最高+10.3%
  • 适合需要精准空间推理的多模态任务

大型视觉语言模型(LVLM)在语义理解上表现优异,但在细粒度空间定位上表现不佳,因模型需隐式推断复杂几何结构而无法生成显式空间解释。本文提出Perceptio,一种具备2D与3D空间推理能力的感知增强型LVLM,通过在自回归序列中显式生成语义分割标记和深度标记实现。具体而言,(i) 从强单目教师模型中蒸馏出VQVAE深度码本,将密集深度压缩为紧凑序列;(ii) 将SAM2生成的语义分割标记与VQ-VAE深度标记融入LLM,使模型先输出空间标记再作答。为稳定深度标记生成,引入复合深度标记目标函数(标记、标记、数量损失)及可微重建的软融合技术。采用跨多样化数据集的多任务协同训练策略,使模型学习感知标记以应对多种下游任务。基于InternVL,Perceptio在多个基准上达到最优性能:在RefCOCO/+/g上参考表达分割的cIoU分别提升+0.8/+1.4/+1.1;空间理解准确率提升10.3%;MMBench准确率提升1.0%,证明显式空间思维链显著增强了LVLM的空间定位能力。

原文摘要 · Abstract (English)

Large Vision Language Models (LVLMs) excel at semantic understanding but struggle with fine grained spatial grounding, as the model must implicitly infer complex geometry without ever producing a spatial interpretation. We present Perceptio, a perception enhanced LVLM with 2D and 3D spatial reasoning abilities, enabled via explicit semantic segmentation tokens and depth tokens generated directly within the autoregressive sequence. Concretely, we (i) distill a VQVAE depth codebook from a strong monocular teacher to tokenize dense depth into compact sequences, and (ii) integrate SAM2 based semantic segmentation tokens and VQ-VAE depth tokens inside the LLM so the model first emits spatial tokens and then answers. To stabilize depth token generation, we introduce novel composite depth-token objectives (marker, token, and count losses) and a soft-merging technique for differentiable reconstruction. We adopt a multi-task co-training strategy across diverse datasets, letting the model learn perception tokens to tackle multiple downstream tasks. Building on InternVL, Perceptio achieves state-of-the-art performance across benchmarks: improving referring expression segmentation by +0.8/+1.4/+1.1 cIoU on RefCOCO/+/g HardBLINK spatial understanding accuracy by 10.3%, and MMBench accuracy by 1.0%, demonstrating that explicit spatial chain-of-thought materially strengthens spatial grounding in LVLMs.

视觉语言模型空间推理深度估计感知增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。