让多模态大模型能主动搜索网页,边查边改,更准更快。
DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Search
- 根据图片局部动态生成图文搜索请求,提升检索精准度。
- 支持多轮迭代优化文本查询,实现自我反思与修正。
- 基于真实网络数据构建训练集,适合需要实时知识的场景。
现实应用中的多模态大语言模型需访问外部知识源,并响应动态变化的真实世界信息,以应对信息查询和知识密集型用户需求。现有方法如检索增强生成(RAG)、搜索代理和配备搜索功能的多模态大模型,常因流程僵化、搜索调用过多及查询构造不佳导致效率低下和效果不佳。为此,我们提出 DeepMMSearch-R1,首个可执行按需、多轮网络搜索并动态生成图像与文本搜索查询的多模态大模型。该模型能基于输入图像的相关区域启动搜索,使图像搜索更有效;同时可依据检索结果迭代调整文本查询,实现自我反思与修正。我们的方法采用两阶段训练流程:冷启动监督微调后接在线强化学习优化。为训练,我们构建了 DeepMMSearchVQA,一个通过自动化管道结合真实网络搜索信息的新型多模态 VQA 数据集。该数据集包含多样化的多跳问题,融合文本与视觉信息,教会模型何时搜索、搜什么、用哪个工具以及如何推理检索内容。我们在多个知识密集型基准上进行大量实验,验证了本方法的优越性。最后,我们分析结果并提供对推进多模态网络搜索具有价值的见解。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) in real-world applications require access to external knowledge sources and must remain responsive to the dynamic and ever-changing real-world information in order to address information-seeking and knowledge-intensive user queries. Existing approaches, such as retrieval augmented generation (RAG) methods, search agents, and search equipped MLLMs, often suffer from rigid pipelines, excessive search calls, and poorly constructed search queries, which result in inefficiencies and suboptimal outcomes. To address these limitations, we present DeepMMSearch-R1, the first multimodal LLM capable of performing on-demand, multi-turn web searches and dynamically crafting queries for both image and text search tools. Specifically, DeepMMSearch-R1 can initiate web searches based on relevant crops of the input image making the image search more effective, and can iteratively adapt text search queries based on retrieved information, thereby enabling self-reflection and self-correction. Our approach relies on a two-stage training pipeline: a cold start supervised finetuning phase followed by an online reinforcement learning optimization. For training, we introduce DeepMMSearchVQA, a novel multimodal VQA dataset created through an automated pipeline intermixed with real-world information from web search tools. This dataset contains diverse, multi-hop queries that integrate textual and visual information, teaching the model when to search, what to search for, which search tool to use and how to reason over the retrieved information. We conduct extensive experiments across a range of knowledge-intensive benchmarks to demonstrate the superiority of our approach. Finally, we analyze the results and provide insights that are valuable for advancing multimodal web-search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。