通过层间一致性聚合，无需训练即可有效减少视觉语言模型的幻觉。
Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models
- 利用前层表示动态构建语义参考，纠正语义偏离的层。
- 在7个模型上提升28.58至42.6点MME分数，POPE准确率提升2.74%。
- 无需重训练或外部知识，适合实际部署和快速集成。
尽管大型视觉语言模型（LVLM）表现出色，但仍易产生与输入图像不一致的幻觉内容。现有无训练幻觉缓解方法常因性能不稳定且对超参数敏感，限制了实际应用。本文提出解码时基于层聚合的一致性机制（DCLA），无需重训练、微调或外部知识库。DCLA通过聚合前层表示构建动态语义参考，用于纠正语义偏离的层，从而增强层间一致性。在七个LVLM及多个基准测试上的实验表明其普适性：在LLaVA1.5-7B上超越标准解码28.58 MME点，在Qwen2.5-VL上提升42.6 MME点，最强设置下POPE准确率提高2.74个百分点。
原文摘要 · Abstract (English)
Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucinations, where generated content is inconsistent with the input image. Existing training-free hallucination mitigation methods often suffer from unstable performance and high sensitivity to hyperparameter settings, which limits their practicality and broader adoption. In this paper, we propose Decoding with Inter-layer Consistency via Layer Aggregation (DCLA), a training-free decoding mechanism that requires no retraining, fine-tuning, or access to external knowledge bases. Specifically, DCLA constructs a dynamic semantic reference by aggregating representations from previous layers and uses it to correct semantically deviated layers, thereby enforcing inter-layer consistency. Experiments across seven LVLMs and multiple benchmarks demonstrate the generality of DCLA: it surpasses standard decoding by 28.58 MME points on LLaVA1.5-7B and 42.6 MME points on Qwen2.5-VL, while improving POPE accuracy by 2.74 percentage points in the strongest setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。