发现视觉模型中关键的视觉证据检索头，提升长文本理解能力。
VERA: Identifying and Leveraging Visual Evidence Retrieval Heads in Long-Context Understanding
- 通过注意力分析定位动态视觉证据检索头，区别于静态文字识别头。
- 遮蔽该头使性能显著下降，证明其对推理至关重要。
- 无需训练，通过不确定度触发显式描述视觉证据，适合长文本任务研究者。
尽管视觉语言模型（VLMs）在文本理解方面表现良好，但在处理长上下文和复杂推理任务时仍面临挑战。本文通过注意力分析深入解析VLMs在长上下文处理中的内部机制，识别出一类关键的视觉证据检索（VER）头——一种稀疏且动态的注意力头，在推理过程中负责定位视觉线索，与静态的OCR头不同。实验表明，这些头是模型性能的关键因素；遮蔽它们会导致性能显著下降。基于此发现，我们提出VEA（视觉证据检索增强）框架，无需训练即可通过检测模型不确定性（即熵值）来触发由VER头关注的视觉证据的显式口头表达。全面实验表明，VEA显著提升了开源VLMs的长上下文理解能力：在Qwen3-VL-8B-Instruct上平均相对提升21.3%，在GLM-4.1V-Thinking上提升20.1%，覆盖五个基准测试。
原文摘要 · Abstract (English)
While Vision-Language Models (VLMs) have shown promise in textual understanding, they face significant challenges when handling long context and complex reasoning tasks. In this paper, we dissect the internal mechanisms governing long-context processing in VLMs to understand their performance bottlenecks. Through the lens of attention analysis, we identify specific Visual Evidence Retrieval (VER) Heads - a sparse, dynamic set of attention heads critical for locating visual cues during reasoning, distinct from static OCR heads. We demonstrate that these heads are causal to model performance; masking them leads to significant degradation. Leveraging this discovery, we propose VERA (Visual Evidence Retrieval Augmentation), a training-free framework that detects model uncertainty (i.e., entropy) to trigger the explicit verbalization of visual evidence attended by VER heads. Comprehensive experiments demonstrate that VERA significantly improves long-context understanding of open-source VLMs: it yields an average relative improvement of 21.3% on Qwen3-VL-8B-Instruct and 20.1% on GLM-4.1V-Thinking across five benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。