通过动态抑制语言先验，有效减少视觉语言模型的虚构物体问题。
NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
- 基于多模态与纯文本输入差异，动态抑制语言解码器的先验影响。
- 在POPE数据集上，使LLaVA-1.5 7B和Qwen-VL 7B准确率提升6.45和7.21。
- 无需训练、即插即用，适用于多种视觉语言模型。
物体虚构是大型视觉语言模型（LVLMs）中的关键问题，即输出包含输入图像中不存在的物体。本研究通过系统实验分析了视觉编码器与语言解码器在生成虚构中的作用。结果表明，物体虚构主要源于语言解码器的强语言先验。基于此，提出无需训练的动态抑制框架NoLan，根据多模态与纯文本输入输出分布差异，动态调整语言先验强度。实验显示，NoLan在多个任务和模型上显著降低虚构现象。例如，在POPE数据集上，对LLaVA-1.5 7B和Qwen-VL 7B的准确率分别提升6.45和7.21。代码已公开于https://github.com/lingfengren/NoLan。
原文摘要 · Abstract (English)
Object hallucination is a critical issue in Large Vision-Language Models (LVLMs), where outputs include objects that do not appear in the input image. A natural question arises from this phenomenon: Which component of the LVLM pipeline primarily contributes to object hallucinations? The vision encoder to perceive visual information, or the language decoder to generate text responses? In this work, we strive to answer this question through designing a systematic experiment to analyze the roles of the vision encoder and the language decoder in hallucination generation. Our observations reveal that object hallucinations are predominantly associated with the strong priors from the language decoder. Based on this finding, we propose a simple and training-free framework, No-Language-Hallucination Decoding, NoLan, which refines the output distribution by dynamically suppressing language priors, modulated based on the output distribution difference between multimodal and text-only inputs. Experimental results demonstrate that NoLan effectively reduces object hallucinations across various LVLMs on different tasks. For instance, NoLan achieves substantial improvements on POPE, enhancing the accuracy of LLaVA-1.5 7B and Qwen-VL 7B by up to 6.45 and 7.21, respectively. The code is publicly available at: https://github.com/lingfengren/NoLan.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。