通过削弱语言模型的先验偏好，减少大视觉语言模型的幻觉问题。
FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models
- 发现模型中关键层的前馈网络是语言先验来源
- 在多个基准上显著降低幻觉率，且不增加推理开销
- 无需训练，适合快速部署到现有模型
尽管大型视觉语言模型（LVLM）表现出色，仍易产生与输入图像不符的幻觉内容。已有研究将原因归于语言先验对视觉输入的主导作用，并采用对比解码方法缓解，但其机制尚未明确。我们分析了各Transformer层的信息流动，发现注意力模块持续整合视觉证据，而关键层的前馈网络（FFN）则是语言先验的源头。这些先验会覆盖视觉证据，导致中间层正确预测偏离最终输出。基于此，我们提出FADE（FFN衰减解码），一种无需训练的方法，通过衰减FFN输出以减轻语言先验主导。在POPE、CHAIR和MME三个基准上，对LLaVA-1.5、mPLUG-Owl2和InstructBLIP的评估表明，FADE有效缓解幻觉，同时保持推理效率。
原文摘要 · Abstract (English)
Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucination, generating content inconsistent with the input image. Recent studies attribute this to the dominance of language priors over visual inputs and employ contrastive decoding methods to mitigate this dominance, but the mechanistic origin remains unexplored. We investigate the information flow through each transformer layer and find that attention modules consistently aggregate visual evidence, while FFN modules at critical layers act as the source of language priors. These priors can override visual evidence, causing correct predictions in intermediate layers to drift toward incorrect outputs. Based on this insight, we propose FADE (FFN Attenuation for DEcoding), a training-free method that attenuates FFN outputs to reduce language-prior dominance. Evaluations on POPE, CHAIR, and MME benchmarks across LLaVA-1.5, mPLUG-Owl2, and InstructBLIP show that FADE effectively mitigates hallucinations while preserving inference efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。