发现多模态大模型推理时视觉注意力分散，提出无需训练的聚焦方法提升准确率。
Deeper Thought, Weaker Aim: Understanding and Mitigating Perceptual Impairment during Reasoning in Multimodal Large Language Models
- 通过熵-聚焦准则选择关键视觉注意力头并重加权
- 在多个基准上显著提升视觉定位与推理准确率
- 适合关注模型可解释性与视觉理解的开发者
多模态大语言模型在长序列推理中常出现感知能力下降，尤其在视觉问答任务中表现不佳。我们发现其根本原因是注意力发散：在多步推理过程中，模型对图像的注意力逐渐分散，偏离与问题相关的区域，导致‘失焦’。通过分析注意力图谱，我们观察到推理提示会显著降低模型对关键图像区域的关注度。进一步发现，模型整体注意力强度与图像内注意力的空间分散程度存在强相关性。基于此，我们提出无需训练的视觉区域引导注意力（VRGA）框架，根据熵-聚焦标准筛选视觉注意力头并重新加权，有效引导模型在推理时聚焦于问题相关区域。在多个视觉-语言基准上的大量实验表明，该方法能有效缓解感知退化，提升视觉定位与推理准确性，并提供对模型如何处理视觉信息的可解释洞察。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) often suffer from perceptual impairments under extended reasoning modes, particularly in visual question answering (VQA) tasks. We identify attention dispersion as the underlying cause: during multi-step reasoning, the model's visual attention becomes scattered and drifts away from question-relevant regions, effectively "losing focus" on the visual input. To better understand this phenomenon, we analyze the attention maps of MLLMs and observe that reasoning prompts significantly reduce attention to regions critical for answering the question. We further find a strong correlation between the model's overall attention on image tokens and the spatial dispersiveness of its attention within the image. Leveraging this insight, we propose a training-free Visual Region-Guided Attention (VRGA) framework that selects visual heads based on an entropy-focus criterion and reweights their attention, effectively guiding the model to focus on question-relevant regions during reasoning. Extensive experiments on vision-language benchmarks demonstrate that our method effectively alleviates perceptual degradation, leading to improvements in visual grounding and reasoning accuracy while providing interpretable insights into how MLLMs process visual information.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。