通过修复注意力分散导致的视觉模糊，有效减少多模态大模型幻觉。
Correcting Visual Blur Induced by Attention Distraction to Reduce Hallucinations: Algorithm and Theory
- 利用跨头注意力增强与动态历史注意力强化，修正注意力分散问题。
- 在多个基准上验证，无需额外训练即显著降低图像描述幻觉率。
- 适合关注模型视觉理解可靠性与生成一致性的研究者使用。
多模态大语言模型常出现物体幻觉，但其背后的视觉感知机制尚不明确。本文揭示，幻觉与人类注意力分散现象密切相关：人类分心时视觉清晰度下降、描述失准；模型中则表现为多头注意力的空间不一致及解码过程中对图像标记注意力的时序衰减。我们进一步提供理论分析，指出注意力分散会增加模型复杂度并削弱分类泛化能力。基于此，提出注意力聚焦改进图像感知方法（AFIP），通过跨头注意力增强和动态历史注意力强化来纠正注意力分散。大量实验在多个基准和模型上验证了该方法的有效性，且无需额外训练。代码已开源：https://github.com/MIKUZ12/AFIP。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) frequently suffer from object hallucinations, yet the visual perceptual mechanism underlying this failure remains poorly understood. In this work, we reveal that hallucinations are strongly associated with a human-like attention distraction phenomenon, where humans under divided focus experience degraded visual clarity and produce inaccurate descriptions, while in models the same mechanism manifests as spatial inconsistency in multi-head attention and temporal fading of attention to image tokens during decoding. We further provide theoretical insights that attention dispersion increases model complexity and degrades classification generalization. Motivated by these findings, we propose an Attention-Focused Approach for Improved Image Perception (AFIP), which corrects attention distraction via cross-head attention enrichment and reinforces visual grounding through dynamic historical attention enhancement. Extensive experiments on multiple benchmarks and models validate the effectiveness of AFIP without additional training. Code is available at: https://github.com/MIKUZ12/AFIP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。