arXiv:2511.05017cs.CVcs.CL2025-11ACL被引 3

通过优化文本嵌入,让视觉语言模型更依赖图像而非语言联想,减少幻觉。

Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings

  • 设计新方法让模型学习与图像匹配的文本嵌入,打破语言主导偏见。
  • 在多个测试集上幻觉率下降,最高提升3.4%(梅林数据集)。
  • 适合关注模型可靠性、避免虚假生成的研究者和开发者。

大型视觉语言模型(LVLMs)中的幻觉问题长期存在,常因多模态推理时视觉信息整合不足所致。主要原因是模型过度依赖文本先验,忽视视觉线索,导致输出虽语言流畅却与图像不符。例如,面对空厨房台面的图片,模型可能虚构出‘水果碗’或‘咖啡杯’。现有多数LVLM将视觉特征拼接至预训练语言模型输入流,并在大规模视觉-语言数据集上训练,但系统分析发现这种策略加剧了语言主导偏差,使注意力偏向文本而弱化视觉内容。为此,我们提出一种简单有效的视觉特征融合方法,促使模型学习与基础语言模型不同的、受视觉信息引导的文本嵌入,实现更均衡的注意力分配。实验结果表明,该方法显著降低幻觉,在多个基准测试中表现优异:MMVP-MLLM提升9.33%,POPE-AOKVQA提升2.99%,梅林(Merlin)提升最高达3.4%,在HallusionBench硬数据集上提升3%。

原文摘要 · Abstract (English)

Hallucinations in Large Vision-Language Models (LVLMs) remain a persistent challenge, often stemming from inadequate integration of visual information during multimodal reasoning. A key cause is the model's over-reliance on textual priors and underutilization of visual cues, leading to outputs that are linguistically fluent but visually inaccurate. For example, given an image of an empty kitchen countertop, an LVLM might hallucinate a "bowl of fruit" or "cup of coffee", relying on language associations rather than visual evidence. Most LVLMs incorporate visual features by appending them to the input stream of a pre-trained LLM and training on large-scale vision-language datasets. Our systematic analysis reveals that this strategy often leads to over-dependence on textual information due to the inherent bias of LLMs towards language-dominant representations. This imbalance skews attention towards the text over visual content, weakening the model's ability to ground outputs in visual inputs. To address this, we propose a simple yet effective visual feature incorporation method that encourages the model to learn visually-informed textual embeddings distinct from those of the base LLM and promotes a more balanced attention distribution. Experimental results across multiple hallucination benchmarks demonstrate that our method significantly reduces hallucinations and fosters more balanced multimodal reasoning. Notably, our approach achieves substantial gains, including +9.33% on MMVP-MLLM, +2.99% on POPE-AOKVQA, up to +3.4% on Merlin, and +3% on the hard-data split of HallusionBench.

视觉语言模型幻觉抑制多模态对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。