arXiv:2605.27243cs.CV2026-05中稿 · EMNLP

发现视觉语言模型中存在能看图的检索头,可精准定位图文混合长序列中的关键信息。

Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models

论文配图:Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models
图 1 · 摘自论文原文
  • 设计多模态检索头检测方法,量化问题词对图文证据的关注度。
  • 仅4.4%-10.2%的注意力头贡献50%检索得分,且其屏蔽导致性能下降超40个百分点。
  • 无需再训练即可用于图文文档排序,显著提升检索准确率,适合长上下文应用。

大型视觉语言模型越来越多依赖长上下文建模来处理文档、小时级视频和长时序智能体轨迹,需在交错的文本与图像中定位相关证据。以往研究使用大语言模型中的检索头,但其基于复制的评估准则不适用于图像证据。本文提出一种多模态检索头检测方法,通过评分问题词对文本或视觉证据的注意力强度。结果显示,多模态检索头具有稀疏性、内在性和因果重要性:仅4.4%-10.2%的注意力头承担50%的正向检索得分质量;遮蔽前5%选中的头会使两个任务在MMLongBench上的性能分别下降42.5和62.3个百分点,而随机遮蔽影响小得多。进一步分析表明,这些头部分跨模态共享,但在各模态内仍具动态性,图像检索头随上下文长度和数据模态变化更显著。无需额外训练,这些头可直接用于排序视觉丰富的文档:在MMDocIR上,Qwen3-VL-8B使用选中头评分使页面检索的Recall@1提升7.7/7.4(宏/微)点,布局检索提升6.3/6.8点,优于现有最强基线。

原文摘要 · Abstract (English)

Large vision-language models increasingly rely on long-context modeling to reason over documents, hour-level videos, and long-horizon agent trajectories, requiring them to locate relevant evidence across interleaved text and images. Prior work has studied this behavior using retrieval heads in large language models, but its copy-based criterion does not directly apply when evidence appears in images. We introduce a multimodal retrieval head detection method that scores attention from question tokens to textual or visual evidence. With this method, we show that multimodal retrieval heads are sparse, intrinsic, and causally important: only 4.4-10.2% of attention heads account for 50% of the positive retrieval-score mass, and masking the top-5% selected heads reduces performance on two tasks from MMLongBench by 42.5 and 62.3 percentage points, while random-head masking is far less damaging. Further analysis shows that these heads are partly shared across modalities yet remain dynamic within each modality, with image retrieval heads changing more than text retrieval heads as context length and haystack modality change. Without further training, we find that these heads can also be used directly to rank visually rich documents: on MMDocIR, Qwen3-VL-8B selected-head scoring improves Recall@1 by 7.7/7.4 macro/micro points for page retrieval and 6.3/6.8 points for layout retrieval over the strongest reported baseline.

视觉语言模型检索头长上下文多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。