通过分析视觉注意力的高频结构，发现并抑制多模态大模型的幻觉生成。
When Looking Is Not Enough: Visual Attention Structure Reveals Hallucination in MLLMs
- 用层间拉普拉斯能量检测注意力高频结构，定位幻觉产生层。
- 在多个基准上降低幻觉率，同时保持模型通用能力。
- 无需训练的解码方法，适合希望提升模型可信度的研究者。
多模态大语言模型（MLLMs）已成为视觉推理与基于图像的问题回答的关键接口，但依然容易产生视觉幻觉——生成内容与图像事实矛盾或提及不存在的对象。一个核心挑战是，幻觉并非总是因缺乏视觉注意导致：模型可能仍对图像标记分配大量注意力，却在内部错误地走向错误答案。本文发现，通过层间拉普拉斯能量衡量的视觉注意力高频结构，可揭示幻觉偏好出现的层级以及真实答案短暂恢复的层级。基于此，我们提出LaSCD（Laplacian-Spectral Contrastive Decoding）——一种无需训练的解码策略，通过拉普拉斯能量选择信息丰富的层级，并以闭式形式重映射下一词的得分。在幻觉检测与通用多模态基准上的实验表明，LaSCD持续减少幻觉，同时保留模型通用能力，凸显其作为可信解码范式的潜力。代码已开源于https://github.com/macovaseas/LaSCD。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have become a key interface for visual reasoning and grounded question answering, yet they remain vulnerable to visual hallucinations, where generated responses contradict image content or mention nonexistent objects. A central challenge is that hallucination is not always caused by a simple lack of visual attention: the model may still assign substantial attention mass to image tokens while internally drifting toward an incorrect answer. In this paper, we show that the high-frequency structure of visual attention, measured by layer-wise Laplacian energy, reveals both the layer where hallucinated preferences emerge and the layer where the ground-truth answer transiently recovers. Building on this finding, we propose LaSCD (Laplacian-Spectral Contrastive Decoding), a training-free decoding strategy that selects informative layers via Laplacian energy and remaps next-token logits in closed form. Experiments on hallucination and general multimodal benchmarks show that LaSCD consistently reduces hallucination while preserving general capabilities, highlighting its potential as a faithful decoding paradigm. The code is available at https://github.com/macovaseas/LaSCD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。