arXiv:2506.20670cs.CVcs.CL2025-06ACL被引 114

让大模型学会智能搜索,减少无效调用,提升真实场景下的知识获取效率。

MMSearch-R1: Incentivizing LMMs to Search

  • 通过强化学习实现端到端多轮搜索,结合图文工具自主判断何时搜索。
  • 在同等规模下超越RAG基线,搜索次数减少30%以上仍保持高性能。
  • 构建了覆盖多样需求的多模态搜索数据集,支持高效行为训练。

大型多模态模型(LMMs)在真实场景中部署需依赖外部知识源,因现实信息复杂且动态变化。现有方法如检索增强生成(RAG)和提示工程搜索代理依赖固定流程,常导致搜索效率低下或过度调用。本文提出MMSearch-R1,首个端到端强化学习框架,使LMM能在真实互联网环境中按需执行多轮搜索。框架整合图像与文本搜索工具,基于结果导向奖励并引入搜索惩罚,指导模型合理决策。为支持训练,我们通过半自动化流程构建了一个多模态搜索视觉问答数据集,涵盖多样化视觉与文本知识需求,并筛选出包含需搜索与无需搜索样本的平衡子集,对塑造高效、按需搜索行为至关重要。在知识密集型与信息查询类视觉问答任务上的大量实验表明,该模型不仅优于同规模的RAG基线,更以更低搜索调用量达到更大RAG模型性能。进一步的实证分析为推进多模态搜索研究提供了可操作洞见。

原文摘要 · Abstract (English)

Robust deployment of large multimodal models (LMMs) in real-world scenarios requires access to external knowledge sources, given the complexity and dynamic nature of real-world information. Existing approaches such as retrieval-augmented generation (RAG) and prompt engineered search agents rely on rigid pipelines, often leading to inefficient or excessive search behaviors. We present MMSearch-R1, the first end-to-end reinforcement learning framework that enables LMMs to perform on-demand, multi-turn search in real-world Internet environments. Our framework integrates both image and text search tools, allowing the model to reason about when and how to invoke them guided by an outcome-based reward with a search penalty. To support training, We collect a multimodal search VQA dataset through a semi-automated pipeline that covers diverse visual and textual knowledge needs and curate a search-balanced subset with both search-required and search-free samples, which proves essential for shaping efficient and on-demand search behavior. Extensive experiments on knowledge-intensive and info-seeking VQA tasks show that our model not only outperforms RAG-based baselines of the same model size, but also matches the performance of a larger RAG-based model while reducing search calls by over 30%. We further analyze key empirical findings to offer actionable insights for advancing research in multimodal search.

多模态搜索强化学习知识获取RAG优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。