发现大模型在多视角图像中易产生幻觉，提出无训练解码方法有效抑制。
Revealing Multi-View Hallucination in Large Vision-Language Models
- 通过注意力掩码生成负对数几率，抑制非目标视角干扰。
- 在4800个样本上验证，性能提升最高达94.8点。
- 适合研究多视角视觉语言模型幻觉问题的开发者与研究人员。
大型视觉语言模型（LVLM）越来越多地用于从不同视角捕获的多视图图像输入。尽管应用广泛，当前的LVLM常因非目标实例或视角的视觉干扰而生成错误回答，这一现象被称为多视角幻觉（MVH）。为系统分析该问题，我们构建了包含4800个问答对的基准测试集MVH-Bench，涵盖跨实例和跨视角两类幻觉。实验表明，MVH在近期的LVLM中普遍存在。为此，我们提出无需训练的解码技术——参考移位对比解码（RSCD），通过注意力掩码生成负对数几率以抑制视觉干扰。在LLaVA-OneVision和Qwen2.5-VL上的实验显示，RSCD相较现有方法性能提升最高达25.7和94.8点，证明其有效性。
原文摘要 · Abstract (English)
Large vision-language models (LVLMs) are increasingly being applied to multi-view image inputs captured from diverse viewpoints. Despite this growing use, current LVLMs often generate incorrect responses due to visual interference from non-target instances or viewpoints, a phenomenon we term multi-view hallucination (MVH). To systematically analyze this problem, we construct MVH-Bench, a benchmark comprising 4.8k question-answer pairs targeting two types of hallucination: cross-instance and cross-view. Empirical results show that MVH is prevalent across recent LVLMs. To address this issue, we propose Reference Shift Contrastive Decoding (RSCD), a training-free decoding technique that suppresses visual interference by generating negative logits through attention masking. Experiments on MVH-Bench with LLaVA-OneVision and Qwen2.5-VL demonstrate that RSCD improves performance by up to 25.7 and 94.8 points over existing hallucination mitigation methods, highlighting the effectiveness of RSCD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。