通过对比视觉编码器浅层与深层特征，减少多模态大模型幻觉。
Mitigating Hallucination in Multimodal LLMs with Layer Contrastive Decoding
- 利用视觉编码器浅层和深层特征生成输出分布并进行对比
- 在两个基准测试中显著优于现有最先进方法
- 适合关注多模态模型可靠性与鲁棒性的研究者
多模态大语言模型（MLLM）展现出强大的感知与推理能力，但常出现幻觉——生成的语言虽通顺，却与输入图像上下文不一致，包括物体、属性和关系的错误。我们提出一种名为层对比解码（LayerCD）的简单方法。该设计基于观察：浅层视觉特征比深层特征更易引发幻觉，因其仅捕捉有偏的低级信息，不足以支持高层推理。因此，LayerCD通过对比来自视觉编码器浅层和深层特征所生成的输出分布，过滤掉幻觉内容。我们在两个幻觉评测基准上进行了广泛实验，结果表明LayerCD显著优于当前最先进的方法。LayerCD代码已公开于https://github.com/maifoundations/LayerCD。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have shown impressive perception and reasoning capabilities, yet they often suffer from hallucinations -- generating outputs that are linguistically coherent but inconsistent with the context of the input image, including inaccuracies in objects, attributes, and relations. To address this challenge, we propose a simple approach called Layer Contrastive Decoding (LayerCD). Our design is motivated by the observation that shallow visual features are much more likely than deep visual features to cause an MLLM to hallucinate as they only capture biased, low-level information that is insufficient for high-level reasoning. Therefore, LayerCD aims to filter out hallucinations by contrasting the output distributions generated from visual features of different levels, specifically those from the shallow and deep layers of the vision encoder, respectively. We conduct extensive experiments on two hallucination benchmarks and show that LayerCD significantly outperforms current state-of-the-art. The code for LayerCD is available at https://github.com/maifoundations/LayerCD .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。