通过引导模型关注多模态交互信息,有效减少视觉语言模型的幻觉问题。
INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling
- 利用多模态交互信息重引导模型生成过程,无需额外数据。
- 在六个基准上平均提升3.4%,显著优于现有解码策略。
- 适合关注模型可靠性与真实性的研究人员和应用开发者。
大型视觉语言模型(LVLM)在实际应用中常出现幻觉,即生成看似合理但与图像内容不符的回答。这与人类认知存在差异:人类会先整合多模态信息,分析模态间交互,再表达理解。我们对主流LVLM进行实验发现,其虽不如人类明显,但具备类似认知行为。基于此,提出无需训练、不依赖额外数据的INTER算法——通过显式引导模型在生成时重新利用多模态交互信息,抑制幻觉。在包含VQA和图像描述在内的六个基准上,相比当前最优解码策略,五种LVLM平均性能提升达3.4%。代码将在论文接收后公开。
原文摘要 · Abstract (English)
Hallucinations in large vision-language models (LVLMs) pose significant challenges for real-world applications, as LVLMs may generate responses that appear plausible yet remain inconsistent with the associated visual content. This issue rarely occurs in human cognition. We argue that this discrepancy arises from humans' ability to effectively leverage multimodal interaction information in data samples. Specifically, humans typically first gather multimodal information, analyze the interactions across modalities for understanding, and then express their understanding through language. Motivated by this observation, we conduct extensive experiments on popular LVLMs and obtained insights that surprisingly reveal human-like, though less pronounced, cognitive behavior of LVLMs on multimodal samples. Building on these findings, we further propose \textbf{INTER}: \textbf{Inter}action Guidance Sampling, a novel training-free algorithm that mitigate hallucinations without requiring additional data. Specifically, INTER explicitly guides LVLMs to effectively reapply their understanding of multimodal interaction information when generating responses, thereby reducing potential hallucinations. On six benchmarks including VQA and image captioning tasks, INTER achieves an average improvement of up to 3.4\% on five LVLMs compared to the state-of-the-art decoding strategy. The code will be released when the paper is accepted.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。