arXiv:2507.00898cs.CVcs.CL2025-07ICCV被引 13

仅用一次干预就能有效减少大模型幻觉,适合实时应用。

ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models

  • 单次推理中通过一层干预增强关键文本信息
  • 在多个基准上优于现有方法,且计算开销极低
  • 无需训练,部署简单,适合实时系统

近年来的大规模视觉语言模型(LVLM)通过文本回答实现了对图像输入的理解与推理,但在多模态任务中仍面临幻觉问题,影响实际部署可靠性。现有缓解方法依赖对比解码,需多次查询,降低响应速度,不适用于实时场景。为此,我们提出ONLY,一种无需训练的解码方法,仅需一次查询和单层干预即可实现高效去幻觉。具体地,利用每个词元的文本-视觉熵比,选择性放大关键文本信息。大量实验表明,ONLY在多个基准上持续优于当前最优方法,实现最小化实现成本与计算开销。代码已开源:https://github.com/zifuwan/ONLY。

原文摘要 · Abstract (English)

Recent Large Vision-Language Models (LVLMs) have introduced a new paradigm for understanding and reasoning about image input through textual responses. Although they have achieved remarkable performance across a range of multi-modal tasks, they face the persistent challenge of hallucination, which introduces practical weaknesses and raises concerns about their reliable deployment in real-world applications. Existing work has explored contrastive decoding approaches to mitigate this issue, where the output of the original LVLM is compared and contrasted with that of a perturbed version. However, these methods require two or more queries that slow down LVLM response generation, making them less suitable for real-time applications. To overcome this limitation, we propose ONLY, a training-free decoding approach that requires only a single query and a one-layer intervention during decoding, enabling efficient real-time deployment. Specifically, we enhance textual outputs by selectively amplifying crucial textual information using a text-to-visual entropy ratio for each token. Extensive experimental results demonstrate that our proposed ONLY consistently outperforms state-of-the-art methods across various benchmarks while requiring minimal implementation effort and computational cost. Code is available at https://github.com/zifuwan/ONLY.

视觉语言模型幻觉抑制高效解码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。