对比解码看似有效,实则多为假象,难以真正减少多模态模型幻觉。
Rethinking CD: A Reproducibility Study and Extension on the Ineffectiveness of Contrastive Decoding at Mitigating Object Hallucinations in MLLMs

- 通过复现与扩展实验,检验对比解码在不同数据集上的表现
- 发现其提升效果多为虚假,未真正增强视觉定位能力
- 适合关注多模态模型可靠性与幻觉问题的研究者
对比解码(CD)被提出作为无需训练的策略,用于缓解多模态大语言模型(MLLMs)中的物体幻觉,在如POPE等基准上报告了性能提升。然而,近期研究质疑这些提升是否反映真实的视觉定位改善。本研究复现并扩展了《性能提升的幻象:为何对比解码无法缓解多模态大模型中的幻觉》的结论,具体验证了CD是否在判别型数据集上引发单向输出分布偏移,并考察其跨数据集的泛化能力。我们还证实自适应合理性约束(APC)在判别与生成型基准上均将采样退化为贪心搜索。进一步实验分析了不同CD策略在生成型数据集上的对数概率分布,提出代理方法并与其对比,探究幻觉信号在专家与新手模型各层间的传播。基于LLaVA与Qwen在MME、POPE和CHAIR上的实验结果,验证了原始结论:CD带来的表面提升常为虚假,且无法稳定转化为更强的视觉定位能力以减少幻觉。这些发现挑战了当前对比解码策略的有效性,推动开发更可靠的幻觉缓解方法。
原文摘要 · Abstract (English)
Contrastive decoding (CD) has been proposed as a training-free strategy for mitigating object hallucinations in multimodal large language models (MLLMs), with reported gains on benchmarks such as POPE. However, recent work has questioned whether these gains reflect genuine improvements in visual grounding. In this study, we reproduce and extend the findings of "The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs." Specifically, we test the claim that CD induces a unidirectional output distribution shift in discriminative datasets and examine its generalizability across datasets. We also verify that the adaptive plausibility constraint (APC) reduces sampling to greedy search on both discriminative and generative benchmarks. Beyond reproduction, we rigorously study the effects of CD across generative and discriminative datasets. We conduct several experiments that provide additional insights: we analyze the logit distributions induced by different CD strategies on generative datasets, propose a proxy method and compare its performance against CD techniques, and investigate how hallucination signals propagate through each layer of the expert and amateur models. Experimental results across MME, POPE, and CHAIR using LLaVA and Qwen validate the original claims and show that the apparent improvements from CD are often spurious and do not consistently translate into stronger visual grounding for reducing hallucinations. These findings challenge the effectiveness of current contrastive decoding strategies and motivate the development of more reliable approaches for mitigating hallucinations in MLLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。