arXiv:2607.22643cs.AIcs.CV2026-07

先推理再检索,让多模态问答更准更可信。

Reason Before You Retrieve: Agentic Planning for Multi-modal RAG

论文配图:Reason Before You Retrieve: Agentic Planning for Multi-modal RAG
图 1 · 摘自论文原文
  • 用智能体规划检索目标和位置,而非直接搜索
  • 在结构化知识图谱中分步检索,准确率显著提升
  • 适合需要可解释性与高精度的复杂多模态任务

多模态检索增强生成(mRAG)旨在利用外部知识回答图文混合问题,但现有系统通常对原始多模态输入进行扁平化证据空间的直接检索,导致两个关键问题:检索目标不明确(问题意图需与视觉指代对象对齐),且搜索空间结构弱,使语义不同的证据在单一全局排序中竞争。为此,我们提出MM-R2,一种多模态智能体检索框架,通过显式建模‘要检索什么’和‘在哪里检索’,实现检索前推理。MM-R2首先从图像-问题对构建意图对齐的检索状态,捕捉信息需求、视觉指代对象和检索约束;随后在结构化KnowledgeMap中,由智能体选择相关检索单元并发出对齐查询。为支持该能力,我们构建了大规模多步检索轨迹数据集MM-R2-Traj,并采用两阶段后训练策略(监督微调+GRPO)。在Infoseek和Encyclopedic VQA数据集上的实验表明,MM-R2显著优于强基线,在答案准确率上提升明显,同时生成更具可解释性和可验证的检索轨迹。

原文摘要 · Abstract (English)

Multimodal retrieval-augmented generation (mRAG) aims to answer image-text queries with external knowledge, but most existing systems still retrieve directly from raw multimodal input over a flat evidence space. This design often struggles with two key challenges: the retrieval target is under-specified because the question intent must be grounded to the correct visual referent, and the search space is weakly structured, forcing semantically distinct evidence to compete in a single global ranking step. We propose MM-R2, a multimodal agentic retrieval framework that reasons before retrieval by explicitly modeling both what to retrieve and where to search. MM-R2 first constructs an intent-grounded retrieval state from the image-question pair, capturing the information need, grounded referent, and retrieval constraints. It then performs retrieval over a structured KnowledgeMap, where the agent selects relevant retrieval units before issuing grounded queries within them. To enable this capability, we build MM-R2-Traj, a large-scale trajectory dataset of multi-step retrieval processes, and adopt a two-stage post-training strategy with supervised fine-tuning and GRPO. Experiments on Infoseek and Encyclopedic VQA datasets show that MM-R2 substantially outperforms strong baselines on answer accuracy while also yielding more interpretable and verifiable retrieval trajectories.

多模态检索智能体知识图谱

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。