大模型可靠性不在注意力图，而在隐藏状态和深层结构。
Where Reliability Lives in Vision-Language Models: A Mechanistic Study of Attention, Hidden States, and Causal Circuits
- 用统一探针分析三类视觉语言模型的注意力、隐藏状态与因果电路。
- 注意力清晰度几乎无法预测答案正确性，但隐藏状态能精准识别可靠性。
- 早期融合模型更鲁棒，晚期融合模型依赖脆弱的局部电路。
主流观点认为视觉语言模型（VLMs）注意力集中即代表可信。我们通过统一的机械探针（VRP）对三类开源模型（LLaVA-1.5、PaliGemma、Qwen2-VL；3-7B参数）进行测试。结果表明：(i) 注意力结构对正确性预测近乎无效（R_pb=0.001），尽管其对特征提取仍为必要（前30%图像块遮蔽导致准确率下降8.2-11.3个百分点，p<0.001）；(ii) 可靠性在计算后期显现：单个隐藏状态线性探针在POPE数据集上达到AUROC>0.95，且自一致性（K=10）是最佳行为预测指标（R_pb=0.43，推理成本高10倍）；(iii) 因果神经元消融揭示架构差异：晚期融合的LLaVA可靠性集中于脆弱的瓶颈层（前5个探针神经元移除导致物体识别准确率下降8.3个百分点），而早期融合的PaliGemma与Qwen2-VL则广泛分布，可承受约50%峰值层隐藏维度损失且性能下降不超过1个百分点。结论：在3-7B VLM中，可靠性应从隐藏状态几何、层间边际形成及稀疏晚层电路中读取，而非注意力图清晰度。
原文摘要 · Abstract (English)
A pervasive intuition holds that vision-language models (VLMs) are most trustworthy when their attention maps look sharp: concentrated attention on the queried region should imply a confident, calibrated answer. We test this Attention-Confidence Assumption directly. We instrument three open-weight VLM families (LLaVA-1.5, PaliGemma, Qwen2-VL; 3-7B parameters) with a unified mechanistic pipeline -- the VLM Reliability Probe (VRP) -- that compares attention structure, generation dynamics, and hidden-state geometry against a single correctness label. Three results emerge. (i) Attention structure is a near-zero predictor of correctness (R_pb(C_k,y)=0.001, 95% CI [-0.034,0.036]; R_pb(H_s,y)=-0.012, [-0.047,0.024] on a pooled n=3,090 split), even though attention remains causally necessary for feature extraction (top-30% patch masking drops accuracy by 8.2-11.3 pp, p<0.001). (ii) Reliability becomes legible later in the computation: a single hidden-state linear probe reaches AUROC>0.95 on POPE for two of three families, and self-consistency at K=10 is the strongest behavioral predictor we measure at 10x inference cost (R_pb=0.43). (iii) Causal neuron-level ablations expose a sharp architectural split with direct monitor-design implications: late-fusion LLaVA concentrates reliability in a fragile late bottleneck (-8.3 pp object-identification accuracy after top-5 probe-neuron ablation), whereas early-fusion PaliGemma and Qwen2-VL distribute it widely and absorb destruction of ~50% of their peak-layer hidden dimension with <=1 pp degradation. The takeaway is narrow but consequential: in 3-7B VLMs, reliability is read more reliably off hidden-state geometry, layer-wise margin formation, and sparse late-layer circuits than off attention-map sharpness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。