VLM模型推理空间关系不依赖精准定位，靠粗略锚点即可完成。
Grounding Isn't Knowing: Do VLMs Need Object Localization for Spatial Reasoning?
- 通过消融与注意力分析发现，模型用粗略目标锚点而非精确框定位
- 位置信息在关系判断前就可解码，仅少数注意力头起关键中介作用
- 适合关注视觉语言模型机制的科研人员阅读
视觉语言模型（VLMs）能回答空间问题，但其从物体定位到空间推理的内在机制仍不清晰。本文研究了LLaVA-1.5与Qwen2.5-VL两类模型，采用标记消融、层级探查、注意力敲除和因果中介分析等工具。结果表明，空间关系预测遵循分阶段的定位到推理过程：物体对齐标记建立粗略的目标-参考锚点，精确边界并非必需。位置信息在关系决策前即可解码，且仅有少量注意力头中介定位与空间推理的因果效应。两任务共享早期定位处理，但最终依赖部分不同的专用路径。实验揭示了模型如何将物体定位转化为空间关系的层次化机制，证明‘知道物体在哪’不等于‘知道它们如何关联’。
原文摘要 · Abstract (English)
Vision-language models (VLMs) can answer spatial questions, yet the mechanisms connecting object grounding to spatial reasoning remain poorly understood. It is underexplored whether spatial reasoning internally requires precise objects localization, or can bypass explicit localization through global layout cues. In this work, we investigate two representative model families, LLaVA-1.5 and Qwen2.5-VL, using a suite of mechanistic interpretability tools, including token ablation, layer-wise probing, attention knockout, and causal mediation analysis. We find that spatial relation prediction follows a staged grounding-to-reasoning process in which object-aligned tokens establish coarse target-reference anchors, while precise bounding-box boundaries are not required. Positional information becomes decodable before relation decisions emerge, and a small set of attention heads mediates the causal effects of both localization and spatial reasoning. The two tasks share early grounding-related processing but ultimately rely on partially distinct specialized pathways. Through rigorous experiments, we provide a token-, layer-, and head-level account of how VLMs transform object grounding into spatial relations, showing that knowing where objects are is not equivalent to knowing how they relate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。