提出双流解码框架,让模型生成更贴近图像的推理结果。
Instruction-Evidence Contrastive Dual-Stream Decoding for Grounded Vision-Language Reasoning
- 双流并行解码:语言驱动流与视觉驱动流并行生成
- 在多个数据集上显著降低幻觉率,提升推理准确率
- 适合需要强视觉接地的生成任务,如视觉问答与描述
视觉语言模型在指令遵循和开放式视觉语言推理中表现优异,但常生成看似流畅却缺乏视觉依据的内容。已有研究表明,指令提示会加剧这一问题,尤其在视觉信号模糊时放大语言先验。为此,本文提出一种解码框架——指令-证据对比双流解码(IECD²),在每一步解码中维护两个并行的词元概率分布:一个由指令驱动,强调表达性与信息量;另一个由视觉证据驱动,严格保证与图像一致。两者通过基于对称KL散度的对比门动态融合,抑制仅依赖语言先验但无图像支持的词元,同时保留二者一致的部分。在POPE、MME、VQAv2、AMBER和MSCOCO等多个数据集上的实验表明,相比现有最优解码方法,该方法在任务准确率与推理性能上均有持续提升,并显著减少幻觉现象。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) exhibit strong performance in instruction following and open-ended vision-language reasoning, yet they frequently generate fluent outputs that are weakly grounded in visual evidence. Prior works have shown that instruction prompting further worsens this issue by amplifying language priors, especially when the visual signal is uncertain or ambiguous. To address this challenge, we propose a decoding framework that explicitly balances linguistic informativeness and visual faithfulness during generation. Our method, Instruction-Evidence Contrastive Dual-Stream Decoding (IECD$^2$), maintains two parallel probability distribution of tokens at each decoding step: an instruction-driven stream that promotes expressive and informative responses, and an evidence-driven stream that enforces strict grounding in the image. These two streams are adaptively fused using a symmetric KL-based contrastive gate, which suppresses tokens favored by language priors but unsupported by visual evidence, while preserving them when both distributions agree. We evaluate IECD$^2$ on multiple datasets spanning various generative vision-language reasoning tasks such as captioning and visual question answering on multiple datasets such as, POPE, MME, VQAv2, AMBER, and MSCOCO. IECD$^2$ demonstrates consistent improvements in task accuracy and reasoning performance with substantial reduction in hallucination compared to state-of-the-art decoding approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。