arXiv:2507.15652cs.CV2025-07

通过提取中间层视觉事实,有效减少多模态大模型幻觉。

Extracting Visual Facts from Intermediate Layers for Mitigating Hallucinations in Multimodal Large Language Models

  • 从中间层动态选取含真实视觉信息的层进行干预。
  • 在多个基准上幻觉率显著降低,优于基线方法。
  • 无需训练、适配性强,适合各类多模态模型使用。

多模态大语言模型(MLLMs)在结合视觉识别与语言理解方面取得显著进展,但依然存在物体幻觉问题,即生成看似合理却与图像事实不符的内容。近期研究发现,模型先验知识会抑制深层中的视觉信息,导致幻觉。然而,这种抑制在中间层如何发生仍不明确。我们观察到,中间层中视觉事实知识与原始/先验概率分布差异的变化趋势相似。受此启发,提出EVA(Decoding by Extracting Visual Facts),一种无需训练的方法:通过对比原图输入与纯文本输入在选定中间层的输出分布,提取视觉事实知识,并按比例融入最终层以修正输出逻辑。EVA具有模型无关性,可无缝集成至多种经典解码策略,适用于不同MLLM。在主流评测集上验证表明,其显著降低幻觉率,证明了其有效性。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) have made significant strides by combining visual recognition and language understanding to generate content that is both coherent and contextually accurate. However, MLLMs continue to struggle with object hallucinations, where models produce seemingly plausible but factually incorrect outputs, including objects that do not exist in the image. Recent work has revealed that the prior knowledge in MLLMs significantly suppresses visual information in deep layers, causing hallucinatory outputs. However, how these priors suppress visual information at the intermediate layer stage in MLLMs remains unclear. We observe that visual factual knowledge and the differences between intermediate-layer prior/original probability distributions show similar evolutionary trends in intermediate layers. Motivated by this, we introduce Decoding by Extracting Visual Facts (EVA), a simple, training-free method that dynamically selects intermediate layers with the most significant visual factual information. By contrasting the output distributions of the selected layer derived from the original input and pure-text input, EVA extracts visual factual knowledge and proportionally incorporates it into the final layer to correct the output logits. Importantly, EVA is model-agnostic, seamlessly integrates with various classic decoding strategies, and is applicable across different MLLMs. We validate EVA on widely-used benchmarks, and the results show that it significantly reduces hallucination rates compared to baseline methods, underscoring its effectiveness in mitigating hallucinations.

多模态模型幻觉抑制视觉事实

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。