视觉语言模型不是看不清，而是没选对答案。
Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts
- 通过层间探针分析发现视觉信号早期已准确编码
- 最终答案错误源于推理阶段的判断失误，非感知缺陷
- 仅修改图像令牌可显著提升正确率，适合改进模型决策
当视觉语言模型看到蓝色香蕉却回答‘黄色’时，问题是感知缺失还是判断失误？我们在十种不同规模的VLM上研究发现：即使回答错误，模型仍能像正确回答者一样强地编码视觉证据。通过多模态仲裁交叉分析（MAC）和逐层对数透镜探针，我们追踪了每一层中视觉与先验信号的竞争关系。早期层中视觉属性可线性解码（AUC > 0.86），且成功与失败样本的编码强度几乎相同。但最终层对数差距（logit gap）与接地结果的相关性高达ρ=0.847，更能预测成败。通过全序列激活修补实验，我们发现标准的最后标记干预无效；而替换由MAC识别层的完整标记序列，可改变60%至84%的输出。部分标记分解显示，只有图像标记具有因果影响，文本标记无作用。通过缩放消除架构差异后，实现完美保留。进一步提出无需训练的激活引导方法，在早期层干预可使视觉接地提升最高+3.8%，但在某些设置下性能下降。结论明确：模型早已‘看见’，问题在于‘选择’——精准干预可弥合这一鸿沟。
原文摘要 · Abstract (English)
When a Vision-Language Model (VLM) sees a blue banana and answers "yellow", is the problem of perception or arbitration? We explore the question in ten VLMs with various sizes and reveal an Encoding-Grounding Dissociation: models that fail to report what they see (and thus provide a wrong answer) still encode the visual evidence as strongly as models that provide the correct answer. Using Multimodal Arbitration Crossover (MAC) analysis with layer-by-layer Logit Lens probing, we track the competition between visual and prior signals across every layer of each model. We show that visual attributes can be linearly decodable from early layers (AUC > 0.86). The accuracy remains nearly identical for both successful and failed samples. However, the gap in the final-layer logit - not the strength of encoding - better predicts grounding outcomes with a correlation of $ρ=$ 0.847. After having studied when VLMs base their answers on image clues rather than prior knowledge, we want to understand the causal relationships. We establish causality through full-sequence activation patching. The standard last-token interventions in LLM interpretability do not affect VLMs. In contrast, replacing the full token sequence at layers identified by MAC alters 60 to 84% of outputs. Partial-token decomposition shows that image tokens carry almost all of the causal impact, while text tokens have none. Scaling addresses the remaining architectural differences to achieve perfect retention. Moving from diagnosis to intervention, we show that training-free activation steering - both linear and sparse autoencoder-guided - in early layers can improve visual grounding by up to +3.8% with degrading performance in some setups. Overall, these findings lead to a clear conclusion: VLMs already see well, but the challenge is acting on what they see. Targeted interventions can help to bridge this gap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。