通过视觉证据签名识别并修正模型幻觉对象,不损伤真实内容。
Hallucinations Leave a Grounding Signature:Verifier-Guided Decoding for Selective Object Correction

- 利用内在接地签名检测生成中每个对象的视觉支持度
- 在AMBER-G数据集上减少43.6%幻觉对象,保留99.6%真实对象覆盖
- 仅对高风险对象干预,保持原模型理解力与回答长度
大型视觉语言模型常在图像中生成不存在的对象。现有方法缺乏可靠的逐对象接地诊断,通常采取粗粒度干预,导致视觉理解受损、回复变短、真实对象覆盖下降。关键挑战在于生成过程中判断每个新出现的对象是否具有可靠视觉依据,从而实现选择性纠正。然而输出置信度反映的是下一个词的合理性而非视觉支持,使语言先验可让不存在的对象显得可信。我们发现缺失的诊断信息编码于内在接地签名(IGS)——一种分布式的带符号注意力模式,即使对自信的幻觉仍具信息量。基于IGS,提出验证器引导解码(VGD),一个轻量级验证器检查每个新出现的对象提及,若判定为高风险,则回滚键值缓存,抑制该对象及其同义词,并重新生成受影响部分。因仅对高风险提及干预,VGD在减少对象幻觉的同时,保持原有视觉理解能力与真实对象覆盖率。在CHAIR和AMBER-G上的实验表明,VGD达到当前最优幻觉削减效果:在@rec90下,使AMBER-G CHAIR幻觉减少43.6%,真实对象覆盖率保持99.6%;在CHAIR-MSCOCO CHAIR$_i$/CHAIR$_s$上分别减少37.0%/30.4%,且未缩短描述。
原文摘要 · Abstract (English)
Large vision-language models (LVLMs) often hallucinate objects that are absent from an image. Despite recent progress, existing mitigation methods still lack reliable object-level grounding diagnostics and therefore tend to apply coarse-grained interventions, which can impair visual understanding, shorten responses, and reduce coverage of genuinely grounded objects. The key challenge is thus to detect, during generation, whether each emerging object mention is supported by reliable visual evidence, so that hallucination can be mitigated selectively. Yet output confidence reflects next-token plausibility rather than visual support, allowing language priors to make absent objects appear certain. We show that the missing diagnostic evidence is encoded in an Intrinsic Grounding Signature (IGS), a distributed signed attention pattern that remains informative for such confident hallucinations. Based on IGS, we propose Verifier-Guided Decoding (VGD), a decoding framework in which a lightweight verifier examines each emerging object mention, rolls back the KV cache when the mention is identified as high risk, suppresses the object and its synonyms, and regenerates the affected continuation. Because VGD intervenes only on object mentions identified as high risk, it reduces object hallucination while preserving the model's original visual understanding and grounded object coverage. Experiments on CHAIR and AMBER-G show that VGD achieves state-of-the-art object hallucination reduction: at @rec90, it cuts AMBER-G CHAIR by 43.6\% while retaining 99.6\% of grounded-object coverage, and reduces CHAIR-MSCOCO CHAIR$_i$/CHAIR$_s$ by 37.0\%/30.4\% without shortening captions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。