揭示视觉语言模型空间感知失效的根源并提出修复方法
Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Models
- 发现视觉标记嵌入过强抑制了位置编码作用
- 通过干预使模型恢复对位置顺序的敏感性
- 适合研究多模态注意力机制与模型可解释性的人
视觉语言模型（VLMs）在物体识别与描述上表现优异，但在空间推理上常失败。我们研究发现，尽管像LLaVA这样的模型具备位置编码和空间丰富的视觉编码特征，却仍严重低估空间线索。分析显示关键问题在于：视觉标记嵌入的范数远大于文本标记，从而压制了语言模型的位置嵌入。为此，我们开发了三种可解释性工具：(1) 位置敏感度指数，量化对标记顺序的依赖；(2) 跨模态平衡，揭示注意力头分配模式；(3) RoPE敏感度探测器，测量对旋转位置编码的依赖。结果表明，视觉标记与系统提示主导注意力。通过针对性干预验证了这一机制理解，并成功预测性恢复位置敏感性。研究揭示了多模态注意力中此前未知的失效模式，并展示可解释性分析如何指导系统性改进。
原文摘要 · Abstract (English)
Vision Language Models (VLMs) excel at identifying and describing objects but often fail at spatial reasoning. We study why VLMs, such as LLaVA, underutilize spatial cues despite having positional encodings and spatially rich vision encoder features. Our analysis reveals a key imbalance: vision token embeddings have much larger norms than text tokens, suppressing LLM's position embedding. To expose this mechanism, we developed three interpretability tools: (1) the Position Sensitivity Index, which quantifies reliance on token order, (2) the Cross Modality Balance, which reveals attention head allocation patterns, and (3) a RoPE Sensitivity probe, which measures dependence on rotary positional embeddings. These tools uncover that vision tokens and system prompts dominate attention. We validated our mechanistic understanding through targeted interventions that predictably restore positional sensitivity. These findings reveal previously unknown failure modes in multimodal attention and demonstrate how interpretability analysis can guide principled improvements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。