提升视觉问答的准确率，让模型能更好利用外部知识推理。
ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering
- 分两阶段检索+批判模型过滤，确保引入的外部信息高质量
- 在Encyclopedic-VQA和InfoSeek上准确率显著优于现有方法
- 适合需要复杂推理和外部知识的视觉问答任务
多模态大语言模型在理解图文视频方面表现优异，常通过视觉问答（VQA）评估。然而，即使最先进的模型在领域特定或知识密集型问题上仍表现不佳，因相关知识在预训练数据中覆盖不足。知识增强型视觉问答（KB-VQA）通过检索外部文档来辅助答案生成，但现有方法存在召回精度低、噪声段落多、推理能力弱等问题。为此，我们提出ReAG，一种新型推理增强型多模态检索增强生成方法，结合粗粒度与细粒度检索，并引入批判模型过滤无关段落，确保高质量上下文。模型采用多阶段训练策略，通过强化学习提升对检索内容的推理能力，监督微调仅作为冷启动。在Encyclopedic-VQA和InfoSeek上的大量实验表明，ReAG显著优于现有方法，提升了答案准确率，并提供基于检索证据的可解释推理。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have shown impressive capabilities in jointly understanding text, images, and videos, often evaluated via Visual Question Answering (VQA). However, even state-of-the-art MLLMs struggle with domain-specific or knowledge-intensive queries, where relevant information is underrepresented in pre-training data. Knowledge-based VQA (KB-VQA) addresses this by retrieving external documents to condition answer generation, but current retrieval-augmented approaches suffer from low precision, noisy passages, and limited reasoning. To address this, we propose ReAG, a novel Reasoning-Augmented Multimodal RAG approach that combines coarse- and fine-grained retrieval with a critic model that filters irrelevant passages, ensuring high-quality additional context. The model follows a multi-stage training strategy leveraging reinforcement learning to enhance reasoning over retrieved content, while supervised fine-tuning serves only as a cold start. Extensive experiments on Encyclopedic-VQA and InfoSeek demonstrate that ReAG significantly outperforms prior methods, improving answer accuracy and providing interpretable reasoning grounded in retrieved evidence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。