通过多图证据聚合,减少视觉幻觉,提升视觉问答准确性
VisRAG2.0: Mitigating Visual Hallucinations via Evidence-Guided Multi-Image Reasoning in Visual Retrieval-Augmented Generation
- 显式收集每张图的问答相关证据,再综合推理
- 在多个基准上平均提升19%,显著减少错误生成
- 适合需要高可靠视觉推理的应用场景
视觉检索增强生成(VRAG)为视觉-语言模型(VLM)引入外部视觉证据,使其在回答视觉相关问题时超越参数化知识。然而,在多图场景下,VLM仍易产生视觉幻觉,难以准确识别与问题相关的证据。现有方法缺乏显式的跨图证据收集过程,且在联合优化感知与推理时信用分配不足。为此,我们提出EVisRAG,一种基于证据引导的多图推理框架:先观察检索图像,逐图记录与问题相关的视觉证据,再基于聚合证据进行推理与答案生成。我们进一步引入RS-GRPO,将奖励信号与不同阶段的词元片段对齐,提升训练稳定性,强化证据定位与推理的联合优化。在多个视觉问答基准上的实验表明,EVisRAG平均比基线VLM提升约19%,同时显著降低视觉幻觉。结果表明,显式证据收集与定向奖励设计能有效提升多图场景下的视觉定位与推理可靠性。代码与数据已公开于https://github.com/OpenBMB/VisRAG
原文摘要 · Abstract (English)
Visual Retrieval-Augmented Generation (VRAG) has emerged as a promising paradigm for equipping Vision-Language Models (VLMs) with external visual evidence, enabling them to go beyond parametric knowledge when answering visually grounded questions. However, in such multi-image settings, VLMs still often suffer from visual hallucinations and struggle to accurately identify the question-relevant evidence needed for reliable reasoning. Existing methods usually lack an explicit cross-image evidence collection process, and also provide limited credit assignment when jointly optimizing perception and reasoning. To address this issue, we propose EVisRAG, an evidence-guided visual retrieval-augmented framework for multi-image reasoning. EVisRAG first observes the retrieved images, records question-relevant visual evidence from each image, and then performs reasoning and answer generation based on the aggregated evidence. We further introduce RS-GRPO, which aligns reward signals with token spans from different stages, improving training stability and strengthening the joint optimization of evidence localization and reasoning. Experiments on multiple visual question answering benchmarks show that EVisRAG consistently outperforms the backbone VLM by an average of about 19\%, while substantially reducing visual hallucinations. These results demonstrate that explicit evidence collection and scoped reward design are effective for improving visual grounding and reasoning reliability in multi-image settings. Codes and data are available at https://github.com/OpenBMB/VisRAG
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。