arXiv:2412.06474cs.CVcs.AI2024-12NeurIPS被引 18

通过不确定性引导的视觉标记丢弃,减少大模型幻觉输出。

Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decoding

论文配图:Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decoding
图 1 · 摘自论文原文
  • 用文本空间投影分解视觉标记不确定性,聚焦感知误差
  • 推理时对高不确定标记做丢弃,提升输出可靠性
  • 适合关注视觉理解可靠性的研究者和应用开发

大型视觉语言模型在多模态任务中表现优异,但容易误读视觉输入,导致幻觉和不可靠输出。本文提出 DROPOUT DECODING,一种新的推理阶段方法,通过量化视觉标记的不确定性并选择性地掩码高不确定标记来改进解码。该方法将每个视觉标记投影到文本空间,分解为随机性和认知性不确定性;重点关注认知不确定性,更有效捕捉感知错误。受丢弃正则化启发,引入不确定性引导的标记丢弃,将丢弃原则应用于输入视觉标记而非模型参数,且在推理阶段执行。通过聚合多个掩码解码上下文的预测,可稳健缓解视觉标记误解释带来的错误。在 CHAIR、THRONE 和 MMBench 等基准测试中,该方法显著降低物体幻觉(OH),并在多种视觉情境下提升输出的可靠性和质量。代码已公开于 https://github.com/kigb/DropoutDecoding。

原文摘要 · Abstract (English)

Large vision-language models (LVLMs) excel at multimodal tasks but are prone to misinterpreting visual inputs, often resulting in hallucinations and unreliable outputs. We present DROPOUT DECODING, a novel inference-time approach that quantifies the uncertainty of visual tokens and selectively masks uncertain tokens to improve decoding. Our method measures the uncertainty of each visual token by projecting it onto the text space and decomposing it into aleatoric and epistemic components. Specifically, we focus on epistemic uncertainty, which captures perception-related errors more effectively. Inspired by dropout regularization, we introduce uncertainty-guided token dropout, which applies the dropout principle to input visual tokens instead of model parameters, and during inference rather than training. By aggregating predictions from an ensemble of masked decoding contexts, we can robustly mitigate errors arising from visual token misinterpretations. Evaluations on benchmarks including CHAIR, THRONE, and MMBench demonstrate that DROPOUT DECODING significantly reduces object hallucinations (OH) and enhances both reliability and quality of LVLM outputs across diverse visual contexts. Code is released at https://github.com/kigb/DropoutDecoding.

视觉语言模型不确定性估计推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。