让大模型学会智能搜索,减少无效调用,提升真实场景下的知识获取效率。
MMSearch-R1: Incentivizing LMMs to Search
- 通过强化学习实现端到端多轮搜索,结合图文工具自主判断何时搜索。
- 在同等规模下超越RAG基线,搜索次数减少30%以上仍保持高性能。
- 构建了覆盖多样需求的多模态搜索数据集,支持高效行为训练。
大型多模态模型(LMMs)在真实场景中部署需依赖外部知识源,因现实信息复杂且动态变化。现有方法如检索增强生成(RAG)和提示工程搜索代理依赖固定流程,常导致搜索效率低下或过度调用。本文提出MMSearch-R1,首个端到端强化学习框架,使LMM能在真实互联网环境中按需执行多轮搜索。框架整合图像与文本搜索工具,基于结果导向奖励并引入搜索惩罚,指导模型合理决策。为支持训练,我们通过半自动化流程构建了一个多模态搜索视觉问答数据集,涵盖多样化视觉与文本知识需求,并筛选出包含需搜索与无需搜索样本的平衡子集,对塑造高效、按需搜索行为至关重要。在知识密集型与信息查询类视觉问答任务上的大量实验表明,该模型不仅优于同规模的RAG基线,更以更低搜索调用量达到更大RAG模型性能。进一步的实证分析为推进多模态搜索研究提供了可操作洞见。
原文摘要 · Abstract (English)
Robust deployment of large multimodal models (LMMs) in real-world scenarios requires access to external knowledge sources, given the complexity and dynamic nature of real-world information. Existing approaches such as retrieval-augmented generation (RAG) and prompt engineered search agents rely on rigid pipelines, often leading to inefficient or excessive search behaviors. We present MMSearch-R1, the first end-to-end reinforcement learning framework that enables LMMs to perform on-demand, multi-turn search in real-world Internet environments. Our framework integrates both image and text search tools, allowing the model to reason about when and how to invoke them guided by an outcome-based reward with a search penalty. To support training, We collect a multimodal search VQA dataset through a semi-automated pipeline that covers diverse visual and textual knowledge needs and curate a search-balanced subset with both search-required and search-free samples, which proves essential for shaping efficient and on-demand search behavior. Extensive experiments on knowledge-intensive and info-seeking VQA tasks show that our model not only outperforms RAG-based baselines of the same model size, but also matches the performance of a larger RAG-based model while reducing search calls by over 30%. We further analyze key empirical findings to offer actionable insights for advancing research in multimodal search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。