通过视觉引导注意力，减少多模态大模型的幻觉问题。
Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attention
- 利用视觉标记语义构建精准定位，引导模型关注相关图像区域。
- 在图像描述生成中动态抑制已描述区域，提升准确性。
- 无需训练、低延迟，适配高效注意力机制，通用性强。
视觉注意力是多模态大模型（MLLMs）理解视觉信息的主要机制，但其定位能力有限常导致幻觉。我们发现，尽管MLLMs能准确提取视觉标记的语义，却未能充分利用这一优势进行后续推理。为此，提出无需训练的视觉引导注意力（VGA）方法：首先利用视觉标记的语义内容构建精确视觉定位，再以此引导模型聚焦于相关视觉区域；在图像描述生成中，VGA还动态抑制已描述区域以进一步优化引导。每个标记仅需一次前向传播，引入极低延迟开销，且完全兼容FlashAttention等高效注意力实现。在多种MLLMs和多个幻觉评估基准上的实验表明，VGA实现了领先的去幻觉性能。进一步分析证实，显式的视觉引导对提升MLLMs的视觉理解能力至关重要。
原文摘要 · Abstract (English)
Visual attention serves as the primary mechanism through which MLLMs interpret visual information; however, its limited localization capability often leads to hallucinations. We observe that although MLLMs can accurately extract visual semantics from visual tokens, they fail to fully leverage this advantage during subsequent inference. To address this limitation, we propose Vision-Guided Attention (VGA), a training-free method that first constructs precise visual grounding by exploiting the semantic content of visual tokens, and then uses this grounding to guide the model's focus toward relevant visual regions. In image captioning, VGA further refines this guidance dynamically during generation by suppressing regions that have already been described. In VGA, each token undergoes only a single forward pass, introducing a negligible latency overhead. In addition, VGA is fully compatible with efficient attention implementations such as FlashAttention. Extensive experiments across diverse MLLMs and multiple hallucination benchmarks demonstrate that VGA achieves state-of-the-art dehallucination performance. Further analysis confirms that explicit visual guidance plays a crucial role in enhancing the visual understanding capabilities of MLLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。