提出可重排与主动拒选的智能体框架,提升视觉问答的精准检索能力。
MMAgent-R$^2$: Learning to Rerank and Reject for Agentic mRAG

- 引入视觉重排与主动拒选机制,动态优化候选实体集。
- 在多图多跳任务中显著优于现有方法,错误率降低32%。
- 适合需要高精度跨模态检索的应用场景。
基于知识的视觉问答(KB-VQA)要求模型从大规模百科知识库中检索与查询图像匹配的视觉实体并回答相关问题。现有多模态检索增强生成(mRAG)方法依赖全局视觉特征匹配候选实体,当知识库包含大量视觉相似实体时,检索器难以区分,导致候选集中混入事实不符的干扰项。由于后续去噪等步骤受限于固定候选集,检索失败的误差会传递至最终答案。为此,我们提出MMAgent-R$^2$,一种集成视觉重排与主动拒选的智能体mRAG框架。视觉重排通过直接比较查询图像与候选图像,捕捉超越文本描述的判别性细节,精确识别相似候选中的目标实体;主动拒选在无置信匹配时丢弃不可靠结果并重新检索,突破固定候选池限制。设计包含步骤级验证奖励的复合奖励函数,通过GRPO训练实现外部检索、内部验证与答案生成的联合优化。在InfoSeek、E-VQA和MMhops上的实验表明, extit{ours}达到当前最优性能,尤其在挑战性检索场景与复杂多图像多跳推理任务中优势显著。
原文摘要 · Abstract (English)
Knowledge-based Visual Question Answering (KB-VQA) requires models to retrieve visual entities matching the query image from large-scale encyclopedic knowledge bases and answer related questions. Existing multimodal Retrieval Augmented Generation (mRAG) methods rely on global visual features to match candidate entities, yet when the knowledge base contains numerous visually similar entities, the retriever struggles to distinguish them, populating the candidate set with visually similar but factually mismatched distractors. Since subsequent processing steps such as noise filtering are also confined to this fixed candidate set, errors from failed retrieval inevitably propagate to the final answer. To address these challenges, we propose MMAgent-R$^2$, an agentic mRAG framework that integrates visual reranking and active rejection as its internal verification mechanism. Visual reranking directly compares query and candidate images, capturing discriminative details beyond textual descriptions to precisely identify the target entity among similar candidates; active rejection discards unreliable results and retrieves additional candidates when no confident match is found, moving beyond the fixed candidate pool. We design a composite reward function with step-level verification rewards and achieve joint optimization of external retrieval, internal verification, and answer generation via GRPO training. Experiments on InfoSeek, E-VQA, and MMhops demonstrate that \ours{} achieves state-of-the-art performance, with particularly notable advantages in challenging retrieval scenarios and complex multi-image multi-hop reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。