通过分析生成过程，用轻量方法减少视觉语言模型的幻觉。
Context-Aware Decoding for Faithful Vision-Language Generation
- 发现真实词比幻觉词更早获得高概率，利用这一规律干预生成
- 在多个基准上显著降低幻觉率，最长生成512个词仍有效
- 无需训练，适配多种模型，适合追求生成准确性的研究者
幻觉——生成与视觉输入不符的内容——仍是大视觉语言模型（LVLMs）在开放任务（如图像描述和视觉推理）中的关键瓶颈。本文探究了驱动幻觉的逐层生成动态，提出一种无需训练的缓解策略。通过Logit Lens分析，发现真实词在最终候选中较早积累概率质量，而幻觉词则滞后。基于此，我们提出上下文嵌入注入（CEI），将最后一个输入词的隐藏状态作为锚定信号，在解码全程保持视觉一致性。在CHAIR、AMBER和MMHal-Bench基准上评估，最大生成长度达512词，CEI在三种主流LVLM上均优于现有基线，其动态变体达到最低整体幻觉率。该工作结合机制洞察与可扩展干预，推动了视觉语言模型幻觉问题的解决。
原文摘要 · Abstract (English)
Hallucinations, generating responses inconsistent with the visual input, remain a critical limitation of large vision-language models (LVLMs), especially in open-ended tasks such as image captioning and visual reasoning. In this work, we probe the layer-wise generation dynamics that drive hallucinations and propose a training-free mitigation strategy. Employing the Logit Lens, we examine how LVLMs construct next-token distributions across decoder layers, uncovering a pronounced commitment-depth gap: truthful tokens accumulate probability mass on their final candidates earlier than hallucinatory ones. Drawing on this discovery, we introduce Context Embedding Injection (CEI), a lightweight method that harnesses the hidden state of the last input token-the context embedding-as a grounding signal to maintain visual fidelity throughout decoding and curb hallucinations. Evaluated on the CHAIR, AMBER, and MMHal-Bench benchmarks (with a maximum token length of 512), CEI outperforms state-of-the-art baselines across three LVLMs, with its dynamic variant yielding the lowest overall hallucination rates. By integrating novel mechanistic insights with a scalable intervention, this work advances the mitigation of hallucinations in LVLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。