构建开放的多模态深度搜索框架,提升AI跨视觉与文本推理能力。
MTA-Agent: An Open Recipe for Multimodal Deep Search Agents
- 设计多跳工具增强代理,自动调用工具检索并验证图文证据
- 生成21000条高质量多跳问答数据,使模型推理步数从2.27升至4.28
- 开源全链路方案,支持低成本复现与未来研究
多模态大语言模型在视觉理解上表现强劲,但在需深度搜索与多步推理的任务中仍受限。本文构建高质量、经验证的多跳视觉-语言训练数据,提出多跳工具增强型证据驱动问答合成代理(MTA-Agent)。该代理可自动选择工具及其参数,从视觉与文本源中检索并验证证据,并生成结构化多跳问答轨迹。基于多样VQA种子数据集,本工作构建大规模训练数据集MTA-Vision-DeepSearch,包含21,000条高质量多跳样本,经多阶段验证确保事实一致性与答案唯一性。使用该数据训练的32B开源多模态搜索代理,在六个挑战性基准上平均达54.63%,优于GPT-5(51.86%)、Gemini-2.5-Pro(50.98%)和Gemini-3-Pro(54.46%),且推理深度与工具使用行为显著提升。此外,通过回放缓存交互可无需实时工具调用完成训练,大幅降低开销。本文完整开放数据集、训练轨迹与实现细节,为多模态深度搜索提供可复现的开源方案。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have demonstrated strong capabilities in visual understanding, yet they remain limited in complex, multi-step reasoning that requires deep searching and integrating visual evidence with external knowledge. In this work, we address this challenge by constructing high-quality, verified multi-hop vision-language training data for multimodal deep-search agents. We propose a Multi-hop Tool-Augmented Agent for Evidence-based QA Synthesis (MTA-Agent), which automatically selects tools and their parameters to retrieve and validate evidence from both visual and textual sources and generates structured multi-hop question-answer trajectories. Starting from diverse VQA seed datasets, our pipeline produces a large-scale training dataset, MTA-Vision-DeepSearch, containing 21K high-quality multi-hop examples. The data is filtered through a multi-stage verification process to ensure factual consistency and answer uniqueness. Using MTA-Vision-DeepSearch, a 32B open-source multimodal search agent achieves state-of-the-art performance, reaching an average of 54.63\% across six challenging benchmarks, outperforming GPT-5 (51.86\%), Gemini-2.5-Pro (50.98\%), and Gemini-3-Pro (54.46\%) under the same tool settings. We further show that training on our data improves both reasoning depth and tool-use behavior, increasing the average number of steps from 2.27 to 4.28, and leading to more systematic and persistent search strategies. Additionally, we demonstrate that training can be performed without real-time tool calls by replaying cached interactions, significantly reducing training cost. Importantly, we present MTA-Agent as a fully open recipe for multimodal deep search: we release the entire dataset, training trajectories, and implementation details to enable reproducibility and future research on open multimodal search agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。