arXiv:2605.01733cs.CVcs.AI2026-05

提出无需训练的可信度机制,让视觉语言模型更智能地判断自动生成描述的可信度。

GEASS: Gated Evidence-Adaptive Selective Caption Trust for Vision-Language Models

论文配图:GEASS: Gated Evidence-Adaptive Selective Caption Trust for Vision-Language Models
图 1 · 摘自论文原文
  • 根据查询内容动态决定是否信任自生成描述,避免错误信息干扰
  • 在两个基准上提升性能,使错误率降低近10个百分点
  • 仅需两次前向传播,不增加参数,可直接部署于现有模型

视觉语言模型常产生不存在的物体幻觉。现有方法尝试将自生成描述作为辅助证据输入模型,但我们发现这种做法反而会降低准确率:在HallusionBench上,Qwen2.5-VL-3B†性能下降近十点。为探究原因,我们构建了GD-Probe诊断集,通过全局与细节问题对比,发现描述的有用性取决于具体查询——同一描述对全局问题有益,对细节问题有害,根源在于其与图像争夺注意力,且效果由描述是否覆盖查询内容决定。关键的是,该机制可通过解码器已输出的量读取,无需访问注意力或定位。我们据此设计GEASS(Gated Evidence-Adaptive Selective Caption Trust),一个无需训练、基于对数概率的模块:根据干净路径置信度进行门控,按熵减少程度加权,并在双路径分歧时提高证据门槛。在四个VLM和两个基准(POPE、HallusionBench)上,GEASS在单一固定设置下优于原始推理与对比解码,仅增加两次前向传播,无额外参数。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) hallucinate objects that are not present, and a growing line of work tries to curb this by feeding the model its own generated caption as auxiliary evidence -- assuming that a caption, once available, is something to consume. We show this fails: naively appending a caption can lower accuracy rather than raise it, dropping Qwen2.5-VL-3B† on HallusionBench by nearly ten points. To understand why, we build GD-Probe, a diagnostic set that pairs a global and a detail question on the same image, so that any difference in caption effect is attributable to the question alone. Caption utility proves to be a per-query property: the same caption helps global questions and harms detail ones, through a single mechanism -- an embedded caption competes with the image for attention and pulls the model's evidence onto its own text -- whose sign is set by whether the caption covers the queried content. Crucially, this regime is readable from quantities the decoder already emits, with no attention access or grounding. We turn this into GEASS (Gated Evidence-Adaptive Selective Caption Trust), a training-free, logit-level module that decides per query how much of the caption to trust, gating it by the clean path's confidence, weighting it by the entropy reduction it induces, and raising the evidence bar when the two pathways disagree. Across four VLMs and two benchmarks (POPE and HallusionBench), GEASS improves over both vanilla inference and contrastive decoding under a single fixed setting, adding only two forward passes and no parameters.

视觉语言模型幻觉抑制自生成描述可信度控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。