arXiv:2506.21509cs.CV2025-06

解决视觉语言模型生成时的语义漂移问题,让输出更贴合图像。

Curing Semantic Drift: A Dynamic Approach to Grounding Generation in Large Vision-Language Models

  • 通过动态校准候选词的视觉相关性,选择更符合图像的词汇。
  • 在多个数据集上显著减少幻觉,且不影响回答质量。
  • 无需训练,适配不同模型和视觉主干,计算开销低。

大型视觉语言模型(LVLMs)在强大语言先验与视觉证据之间面临矛盾,常出现语义漂移:生成内容逐渐脱离输入图像,且可能在特定解码步骤突然发生。通过逐标记诊断发现,幻觉并非因缺乏合理候选词,而是选择失败——模型即使存在更符合图像的选项,仍会选择语言上顺手但视觉不符的词。为此,我们提出无需训练的动态逻辑校准(DLC)框架,引入轻量级视觉裁判,在漂移发生时精准干预。每一步对前k个候选词进行双重视觉验证:(1)候选词的内在视觉相关性,(2)其上下文视觉一致性。结合自适应历史基线计算相对视觉优势,动态调整逻辑值,优先选择视觉可信的词。在CHAIR、POPE、SHR、GPT-4o评估及MME上的大量实验表明,DLC能持续降低多种LVLM的幻觉率,同时保持输出质量。进一步分析验证了对不同视觉主干的鲁棒性,并展示了在候选池大小变化下的质量与成本良好权衡。代码将发布于https://github.com/JiaheChen2002/DLC。

原文摘要 · Abstract (English)

Large Vision-Language Models (LVLMs) face a tug-of-war between powerful linguistic priors and visual evidence, often leading to \emph{semantic drift}: a progressive detachment from the input image that can abruptly emerge at specific decoding steps. Through a token-level diagnosis, we show that hallucination is frequently triggered not by the absence of grounded candidates, but by a failure of selection -- the model chooses a linguistically convenient yet visually unfaithful token even when better grounded alternatives exist. Motivated by this insight, we propose \textbf{D}ynamic \textbf{L}ogits \textbf{C}alibration (DLC), a training-free decoding framework that introduces a lightweight visual referee to intervene exactly when drift happens. At each step, DLC performs a dual-aspect grounding check on top-$k$ candidates: (1) it assesses the intrinsic visual relevance of a candidate token and (2) its contextual visual coherence. These signals are evaluated against an adaptive historical baseline to compute a relative visual advantage, which is then used to dynamically calibrate logits and favor grounded tokens. Extensive experiments on CHAIR, POPE, SHR, GPT-4o evaluation, and MME demonstrate that DLC consistently reduces hallucinations across multiple LVLMs while preserving response quality. Further analyses validate robustness to different vision backbones and demonstrate a favorable trade-off between output quality and computational cost as the candidate pool size varies. Code will be released on https://github.com/JiaheChen2002/DLC.

视觉语言模型幻觉抑制动态校准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。