首个支持长链推理的多模态智能搜索评测基准,提升模型跨模态推理能力。
MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains
- 构建包含3.7步推理链的多模态智能搜索数据集,标注子问题与检索模态。
- 发现主流模型存在过度或不足检索、模态错配等问题,平均准确率不足60%。
- 提出Search-Align框架,通过过程监督微调,显著提升开源模型规划与检索质量。
随着对分步式、跨模态与知识驱动推理需求的增长,多模态大语言模型(MLLM)正从传统的固定检索-生成范式演进为更复杂的代理式多模态检索增强生成(MM-RAG)。现有基准主要聚焦于简化的问答任务与短链检索,忽视了自适应规划与多模态推理。我们提出MC-Search,首个具备长且分步标注推理链的代理式MM-RAG基准,涵盖五种典型推理结构。每条样本明确包含子问题、检索模态、支持事实与中间答案,通过HAVE(逐跳证据归属与验证)确保真实性,共生成3,333个高质量样本,平均3.7跳。除答案准确性外,引入过程级度量评估推理质量、步骤化检索与规划准确率。通过构建统一的代理式MM-RAG流水线,我们对六种领先MLLM进行评测,揭示系统性问题如过度/不足检索与模态错配。最后提出Search-Align,一种利用验证推理链的过程监督微调框架,表明该数据不仅支持可信评估,还能有效提升开源MLLM的规划与检索保真度。
原文摘要 · Abstract (English)
With the increasing demand for step-wise, cross-modal, and knowledge-grounded reasoning, multimodal large language models (MLLMs) are evolving beyond the traditional fixed retrieve-then-generate paradigm toward more sophisticated agentic multimodal retrieval-augmented generation (MM-RAG). Existing benchmarks, however, mainly focus on simplified QA with short retrieval chains, leaving adaptive planning and multimodal reasoning underexplored. We present MC-Search, the first benchmark for agentic MM-RAG with long, step-wise annotated reasoning chains spanning five representative reasoning structures. Each example specifies sub-questions, retrieval modalities, supporting facts, and intermediate answers, with fidelity ensured by HAVE (Hop-wise Attribution and Verification of Evidence), resulting in 3,333 high-quality examples averaging 3.7 hops. Beyond answer accuracy, MC-Search introduces new process-level metrics for reasoning quality, stepwise retrieval and planning accuracy. By developing a unified agentic MM-RAG pipeline, we benchmark six leading MLLMs and reveal systematic issues such as over- and under-retrieval and modality-misaligned planning. Finally, we introduce Search-Align, a process-supervised fine-tuning framework leveraging verified reasoning chains, showing that our data not only enables faithful evaluation but also improves planning and retrieval fidelity in open-source MLLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。