用视觉噪声抑制大模型幻觉，不改模型也能提升准确性
Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs
- 通过优化视觉噪声干扰，让模型更贴近真实图像
- 在8个主流视觉语言模型上均显著降低幻觉率
- 适合关注模型可靠性与事实准确性的研究者
大型视觉语言模型（LVM）将视觉感知能力引入大语言模型（LLM），使其能处理和理解视觉信息。但其可靠性常受对象幻觉问题影响，即模型生成看似合理却与图像事实不符的内容。本文提出一种新型视觉对抗扰动（VAP）方法，通过施加经过精心优化的视觉噪声来缓解幻觉问题，且无需修改基础模型。该方法将幻觉抑制建模为优化问题，利用对抗策略生成有益的视觉扰动，增强模型的事实依据性并减少参数知识偏差。大量实验证明，该方法在8个先进LVM上持续有效降低了对象幻觉，在多种评估中表现稳健。
原文摘要 · Abstract (English)
Large vision-language models (LVMs) extend large language models (LLMs) with visual perception capabilities, enabling them to process and interpret visual information. A major challenge compromising their reliability is object hallucination that LVMs may generate plausible but factually inaccurate information. We propose a novel visual adversarial perturbation (VAP) method to mitigate this hallucination issue. VAP alleviates LVM hallucination by applying strategically optimized visual noise without altering the base model. Our approach formulates hallucination suppression as an optimization problem, leveraging adversarial strategies to generate beneficial visual perturbations that enhance the model's factual grounding and reduce parametric knowledge bias. Extensive experimental results demonstrate that our method consistently reduces object hallucinations across 8 state-of-the-art LVMs, validating its efficacy across diverse evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。