通过打乱视觉结构抑制模型对纹理的依赖，减少大模型幻觉
SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Models
- 引入结构打乱对比解码，通过随机打乱图像块顺序来削弱局部纹理偏好
- 在多个基准上显著降低幻觉率，提升模型对真实物体的识别准确率
- 无需训练即可部署，适合希望快速优化现有视觉语言模型的开发者
大型视觉语言模型在多模态理解与推理方面取得显著进展，但物体幻觉仍是关键挑战。现有研究多关注语言先验或高层统计偏差，却忽视了视觉编码过程中的内在复杂性。我们发现，由于视觉编码器在弱结构监督下呈现‘图像块集合’行为，导致视觉统计偏差，使模型更依赖单个图像块的局部纹理特征而非整体几何结构。这种倾向会引发虚假视觉置信度，从而造成幻觉。为此，我们提出一种无需训练的算法——结构打乱对比解码（SDCD），通过引入结构打乱的视图对输出分布进行对比校准。该方法惩罚在无结构视图下仍保持高置信度的词元，有效抑制了由纹理驱动的偏差。实验表明，SDCD在多个基准上显著缓解了幻觉问题，并提升了大型视觉语言模型的整体多模态能力。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) demonstrate significant progress in multimodal understanding and reasoning, yet object hallucination remains a critical challenge. While existing research focuses on mitigating language priors or high-level statistical biases, they often overlook the internal complexities of the visual encoding process. We identify that visual statistical bias, arising from the inherent Bag-of-Patches behavior of Vision Encoders under weak structural supervision, acts as a contributing factor of object hallucinations. Under this bias, models prioritize local texture features within individual patches over holistic geometric structures. This tendency may induce spurious visual confidence and result in hallucinations. To address this, we introduce a training-free algorithm called Structure-Disrupted Contrastive Decoding (SDCD), which performs contrastive calibration of the output distribution by introducing a shuffled structure-disrupted view. By penalizing tokens that maintain high confidence under this structure-less view, SDCD effectively suppresses the texture-driven bias. Experimental results demonstrate that SDCD significantly mitigates hallucinations across multiple benchmarks and enhances the overall multimodal capabilities of LVLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。