通过遮蔽图像头提升视觉语言模型准确性，减少幻觉。
MaskCD: Mitigating LVLM Hallucinations by Image Head Masked Contrastive Decoding
- 用遮蔽图像头生成对比样本，改进对比解码
- 在多个基准上降低幻觉率，保持模型通用能力
- 适合关注多模态模型可靠性与鲁棒性的研究者
大型视觉语言模型（LVLMs）在下游多模态任务中表现出色，但幻觉问题日益突出，即模型生成内容与输入图文信息矛盾。现有方法如对比解码和注意力调控存在构造对比样本困难或稳定性差的问题。本文提出图像头掩码对比解码（MaskCD），利用LVLM中的“图像头”进行掩码，生成有效的对比样本用于对比解码。在LLaVA-1.5-7b和Qwen-VL-7b上评估，使用CHAIR、POPE、AMBER和MME等多个基准，结果表明MaskCD能有效缓解幻觉现象，同时保持模型的通用性能。相关代码与资源见：https://github.com/Deng-Jingyuan/MaskCD。
原文摘要 · Abstract (English)
Large vision-language models (LVLMs) have shown remarkable performance in visual-language understanding for downstream multimodal tasks. While their capabilities are improving, problems emerge simultaneously. Among those problems, the hallucinations have attracted much attention, which stands for the phenomenon where LVLMs generate contradictory content to their input visual and text contents. Many approaches have been proposed to deal with this issue, such as contrastive decoding and attention manipulation. However, contrastive decoding methods struggle in constructing appropriate contrastive samples, and attention manipulation methods are highly sensitive, lacking stability. In this work, we propose image head Masked Contrastive Decoding (MaskCD). Our approach utilizes the "image heads" in LVLMs, masking them to construct contrastive samples for contrastive decoding. We evaluated MaskCD on LLaVA-1.5-7b and Qwen-VL-7b, using various benchmarks such as CHAIR, POPE, AMBER and MME. The results demonstrate that MaskCD effectively alleviates the phenomenon of hallucinations and retains the general capabilities of LVLMs. Corresponding resources could be found at: https://github.com/Deng-Jingyuan/MaskCD .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。