构建可跨模态搜索的智能研究代理,解决数据与成本难题。
MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baseline
- 用超图生成多模态搜索问答对,连接图文信息
- 分工具优化专家模型,树搜索合成有效搜索路径
- 离线模拟搜索环境,降低训练成本,适合研究型AI开发
我们致力于构建具备显式推理、规划与多工具调用能力的多模态研究代理,以完成深度研究任务。然而,当前面临三大挑战:(1) 搜索密集型多模态问答数据稀缺;(2) 缺乏高效搜索轨迹;(3) 在线搜索API训练成本过高。为此,我们提出Hyper-Search,一种基于超图的QA生成方法,通过建模并连接跨模态的视觉与文本节点,生成需调用多种搜索工具才能解答的多模态问答对。其次,引入DR-TTS,将搜索任务按工具类型分类,分别优化专用工具专家,并通过树搜索重组专家以联合探索搜索轨迹,成功解决复杂任务。第三,构建支持多工具的离线搜索引擎,实现无需使用昂贵在线API的代理强化学习。结合三项设计,我们开发了MM-DeepResearch,一个强大的多模态深度研究代理,在多个基准测试中表现优异。代码已开源。
原文摘要 · Abstract (English)
We aim to develop a multimodal research agent capable of explicit reasoning and planning, multi-tool invocation, and cross-modal information synthesis, enabling it to conduct deep research tasks. However, we observe three main challenges in developing such agents: (1) scarcity of search-intensive multimodal QA data, (2) lack of effective search trajectories, and (3) prohibitive cost of training with online search APIs. To tackle them, we first propose Hyper-Search, a hypergraph-based QA generation method that models and connects visual and textual nodes within and across modalities, enabling to generate search-intensive multimodal QA pairs that require invoking various search tools to solve. Second, we introduce DR-TTS, which first decomposes search-involved tasks into several categories according to search tool types, and respectively optimize specialized search tool experts for each tool. It then recomposes tool experts to jointly explore search trajectories via tree search, producing trajectories that successfully solve complex tasks using various search tools. Third, we build an offline search engine supporting multiple search tools, enabling agentic reinforcement learning without using costly online search APIs. With the three designs, we develop MM-DeepResearch, a powerful multimodal deep research agent, and extensive results shows its superiority across benchmarks. Code is available at https://github.com/HJYao00/MM-DeepResearch
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。