用模型自身幻觉反制幻觉，提升视觉语言模型准确性
Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decoding
- 通过对比内部表示扰动，动态修正输出分布
- 在POPE和MME数据集上分别提升9%和8%准确率
- 无需训练或外部数据，适合部署阶段优化
大型视觉语言模型（LVLM）在多模态任务中表现优异，但易产生物体幻觉。以往方法依赖监督微调或外部知识，成本高。本文提出一种无需训练的模型无关新方法——内部事实对比解码（IFCD），利用模型自身幻觉特性，在推理阶段抑制幻觉。实验发现，内部表征扰动会加剧语言偏见导致的幻觉。IFCD通过对比扰动分布，校准输出，有效移除幻觉逻辑。结果表明，IFCD显著缓解物体与属性级幻觉，在POPE上平均提升9%准确率，在MME物体幻觉子集上提升8%，优于直接解码。
原文摘要 · Abstract (English)
Large Visual Language Models (LVLMs) integrate visual and linguistic modalities, exhibiting exceptional performance across various multimodal tasks. Nevertheless, LVLMs remain vulnerable to the issue of object hallucinations. Previous efforts to mitigate this issue focus on supervised fine-tuning (SFT) or incorporating external knowledge, both of which entail significant costs related to training and the acquisition of external data. To address these challenges, we propose a novel model-agnostic approach termed Internal Fact-based Contrastive Decoding (IFCD), designed to mitigate and suppress hallucinations during the inference process of LVLMs by exploiting the LVLMs' own hallucinations. IFCD is grounded in experimental observations that alterations to the LVLMs' internal representations tend to amplify hallucinations caused by language bias. By contrasting disturbed distribution, IFCD calibrates the LVLMs' output and effectively removes the hallucinatory logits from the final predictions. Experimental results validate that IFCD significantly alleviates both object-level and attribute-level hallucinations while achieving an average 9% accuracy improvement on POPE and 8% accuracy improvement on MME object hallucinations subset compared with direct decoding, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。