arXiv:2604.03556cs.CVcs.AI2026-04

通过关注阶段抑制低注意力令牌,轻量级降低视觉语言模型幻觉。

Focus Matters: Phase-Aware Suppression for Hallucination in Vision-Language Models

  • 发现视觉编码器处理分三阶段:扩散、聚焦、再扩散。
  • 聚焦阶段低注意力令牌与幻觉高度相关,针对性抑制可降幻觉。
  • 无需训练,单次前向传播即可完成干预,延迟几乎无增加。

大型视觉-语言模型在多模态推理中取得显著进展,但仍易产生物体幻觉,即生成输入图像中不存在的物体描述。现有方法尝试通过抑制视觉编码器中的不可靠视觉信号来缓解幻觉,但许多依赖于每个输入的迭代优化,导致推理延迟大幅增加。本文研究了视觉编码器在大型视觉-语言模型中的内部注意力动态,发现视觉信息处理存在一致的三阶段结构:扩散、聚焦、再扩散。分析表明,幻觉行为对聚焦阶段注意力较低的令牌尤为敏感。基于此观察,我们提出一种轻量级推理时干预方法,在聚焦阶段选择性抑制这些令牌。该方法无需训练,仅通过一次前向传播统计实现,并利用行列式点过程(DPP)在过滤冗余令牌的同时保留多样化的视觉线索。在多个大型视觉-语言模型骨干网络和解码策略上的大量实验表明,该方法持续降低幻觉指标,同时保持出色的描述质量。此外,相比对抗性不确定性估计方法,本方法在幻觉缓解效果相当的前提下,额外推理延迟可忽略不计。

原文摘要 · Abstract (English)

Large Vision-Language Models (LVLMs) have achieved impressive progress in multimodal reasoning, yet they remain prone to object hallucinations, generating descriptions of objects that are not present in the input image. Recent approaches attempt to mitigate hallucinations by suppressing unreliable visual signals in the vision encoder, but many rely on iterative optimization for each input, resulting in substantial inference latency. In this work, we investigate the internal attention dynamics of vision encoders in LVLMs and identify a consistent three-phase structure of visual information processing: diffusion, focus, and rediffusion. Our analysis reveals that hallucination behavior is particularly sensitive to tokens receiving low attention during the focus phase. Motivated by this observation, we propose a lightweight inference-time intervention that selectively suppresses such tokens during the focus phase. The method operates in a training-free manner using statistics from a single forward pass and employs a Determinantal Point Process (DPP) to preserve diverse visual cues while filtering redundant tokens. Extensive experiments across multiple LVLM backbones and decoding strategies demonstrate that the proposed approach consistently reduces hallucination metrics while maintaining competitive caption quality. Moreover, compared to adversarial uncertainty estimation methods, our approach achieves comparable hallucination mitigation with negligible additional inference latency.

视觉语言模型幻觉抑制注意力机制推理加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。