通过概率幻觉检测，高效抑制视觉语言模型的幻觉输出。
Efficient Contrastive Decoding with Probabilistic Hallucination Detection - Mitigating Hallucinations in Large Vision Language Models -
- 推理时对比令牌概率与幻觉评分，修正输出分布
- 在多个数据集上显著降低幻觉率，且速度更快
- 无需重新训练，适配任意开源视觉语言模型
尽管大型视觉语言模型（LVLMs）取得了进展，但其仍会生成与输入图像不符的幻觉响应。为缓解此问题，我们提出高效对比解码（ECD），一种在推理时利用概率幻觉检测将输出分布调整为更符合上下文的简单方法。通过对比令牌概率与幻觉得分，ECD从原始分布中减去幻觉概念，有效抑制幻觉。该方法可应用于任意开源LVLM，无需额外训练。我们在多个基准数据集和不同LVLM上评估，实验表明ECD在减少幻觉方面优于现有先进方法，同时在性能与计算时间上表现更优。
原文摘要 · Abstract (English)
Despite recent advances in Large Vision Language Models (LVLMs), these models still suffer from generating hallucinatory responses that do not align with the visual input provided. To mitigate such hallucinations, we introduce Efficient Contrastive Decoding (ECD), a simple method that leverages probabilistic hallucination detection to shift the output distribution towards contextually accurate answers at inference time. By contrasting token probabilities and hallucination scores, ECD subtracts hallucinated concepts from the original distribution, effectively suppressing hallucinations. Notably, our proposed method can be applied to any open-source LVLM and does not require additional LVLM training. We evaluate our method on several benchmark datasets and across different LVLMs. Our experiments show that ECD effectively mitigates hallucinations, outperforming state-of-the-art methods with respect to performance on LVLM benchmarks and computation time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。