分析视觉语言模型幻觉成因，提出实时抑制幻觉的解码方法。
Exploring Causes and Mitigation of Hallucinations in Large Vision Language Models
- 通过对比有无图像输入时的隐藏层特征，识别幻觉生成的关键词。
- 在图像描述任务中，将幻觉率降低37%，且不损失语义质量。
- 适合关注多模态模型可靠性与可解释性的研究人员使用。
大型视觉语言模型（LVLM）将图像编码器与大语言模型结合，处理多模态输入并完成复杂视觉任务。然而，它们常会生成虚构对象或属性，导致输出不可靠。本研究分析了图像描述任务中的幻觉模式，发现并非所有生成词元都受图像输入影响，且图像依赖性可作为幻觉检测的有效信号。为此，我们构建了一个自动化管道，通过并行推理（带/不带图像输入）提取隐藏表示，训练一个词元级分类器以识别幻觉对象。基于该分类器，我们提出一种推理阶段的解码策略，能有效控制图像描述中的幻觉率。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) integrate image encoders with Large Language Models (LLMs) to process multi-modal inputs and perform complex visual tasks. However, they often generate hallucinations by describing non-existent objects or attributes, compromising their reliability. This study analyzes hallucination patterns in image captioning, showing that not all tokens in the generation process are influenced by image input and that image dependency can serve as a useful signal for hallucination detection. To address this, we develop an automated pipeline to identify hallucinated objects and train a token-level classifier using hidden representations from parallel inference passes-with and without image input. Leveraging this classifier, we introduce a decoding strategy that effectively controls hallucination rates in image captioning at inference time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。