arXiv:2608.07302cs.CVcs.AI2026-08被引 3

用对注意力的解码发现:模型错认物体不是因为没注意,而是看错了。

Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination

论文配图:Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination
图 1 · 摘自论文原文
  • 通过Logit-Lens分析高注意力区域,发现真物与幻觉物都受强关注
  • 提出检测-缓解框架,在多个基准上达到最新性能
  • 适合研究视觉语言模型幻觉问题的学者和开发者

大型视觉-语言模型(LVLMs)常产生图像中并不存在的物体。以往研究多归因于视觉注意力不足,但我们发现真实与幻觉物体在模型中后层均获得同等强度的视觉注意力,表明问题可能不在于注意力强度,而在于注意力对象及其原因。我们使用Logit Lens解码高注意力区域的视觉特征,发现真实物体对应的区域可准确解码为对应目标词元,而幻觉物体则不能。基于此,我们识别出两种幻觉机制:(i) 视觉不确定性,由语义相似或易混淆区域触发;遮蔽这些区域可消除幻觉。(ii) 上下文先验,由强共现先验引发;即使初始注意力区域被遮蔽,幻觉仍持续,注意力会漂移到其他区域。据此,我们提出无需训练的检测-缓解框架:利用Logit-Lens一致性检查检测幻觉,并采用针对性策略——针对视觉不确定性使用高注意力区域遮蔽(HARM),针对上下文先验使用视觉证据增强解码(VEED)。该方法在多个幻觉检测基准上取得当前最佳表现。代码将公开。

原文摘要 · Abstract (English)

Large Vision-Language Models (LVLMs) often suffer from object hallucination, generating objects that are absent from the image. Prior work largely attributes this to insufficient visual attention. However, we find that both real and hallucinated objects receive equally strong visual attention in the model's mid-to-late layers, suggesting that the key issue may not be how much the model attends, but what it attends to and why. To this end, we decode the visual features of high-attention regions using Logit Lens, and observe that regions corresponding to real objects can be correctly decoded to the target object tokens, whereas those for hallucinated objects cannot. Building on this, we identify two hallucination mechanisms: (i) visual uncertainty, triggered by semantically similar or confusable regions; masking these regions eliminates the hallucination. (ii) contextual prior, triggered by strong co-occurrence priors; even when the initially attended region is masked, the hallucination persists and attention drifts to other regions. Based on these findings, we propose a simple yet effective training-free Detect-Mitigate framework comprising a Logit-Lens Consistency Check to detect hallucination and targeted remedies: High-Attention Regions Masking (HARM) for visual uncertainty hallucination, and Visual Evidence Enhanced Decoding (VEED) for contextual prior hallucination. Our approach achieves state-of-the-art results on multiple hallucination benchmarks. Code will be available.

幻觉检测视觉注意力多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。