arXiv:2605.05185cs.CV2026-05被引 19

开源全流程训练框架,让普通研究者也能复现顶尖多模态搜索智能体。

OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents

论文配图:OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents
图 1 · 摘自论文原文
  • 构建高质量训练数据流水线,避免检索捷径和一步到位问题。
  • 在7个基准上平均性能提升超10分,部分任务媲美闭源商业模型。
  • 适合研究多模态智能体、强化学习与开放世界搜索的开发者。

深度搜索已成为前沿多模态智能体的核心能力,使模型能通过主动搜索、证据验证和多步推理解决复杂问题。尽管进展迅速,顶级多模态搜索智能体仍难以复现,主要因缺乏开源高质量训练数据、透明轨迹生成流程或详细训练方案。为此,我们提出OpenSearch-VL,一个基于智能体强化学习的全开源多模态深度搜索智能体训练方案。首先,设计专用数据构建流水线,通过维基百科路径采样、模糊实体重写和源锚视觉定位,联合降低捷径依赖和单步检索崩溃。基于此,构建两个数据集:SearchVL-SFT-36k(用于监督微调)和SearchVL-RL-8k(用于强化学习)。此外,设计多样化工具环境,集成文本搜索、图像搜索、OCR、裁剪、锐化、超分辨率与视角校正,支持智能体结合主动感知与外部知识获取。最后,提出多轮致命错误感知的GRPO训练算法,通过屏蔽故障后令牌并保留故障前有效推理,实现单侧优势钳制。基于该方案,OpenSearch-VL在7个基准上平均性能提升超10点,多个任务表现接近闭源商业模型。我们将开源全部数据、代码与模型,推动多模态深度搜索的开放研究。

原文摘要 · Abstract (English)

Deep search has become a crucial capability for frontier multimodal agents, enabling models to solve complex questions through active search, evidence verification, and multi-step reasoning. Despite rapid progress, top-tier multimodal search agents remain difficult to reproduce, largely due to the absence of open high-quality training data, transparent trajectory synthesis pipelines, or detailed training recipes. To this end, we introduce OpenSearch-VL, a fully open-source recipe for training frontier multimodal deep search agents with agentic reinforcement learning. First, we curated a dedicated pipeline to construct high-quality training data through Wikipedia path sampling, fuzzy entity rewriting, and source-anchor visual grounding, which jointly reduce shortcuts and one-step retrieval collapse. Based on this pipeline, we curate two training datasets, SearchVL-SFT-36k for SFT and SearchVL-RL-8k for RL. Besides, we design a diverse tool environment that unifies text search, image search, OCR, cropping, sharpening, super-resolution, and perspective correction, enabling agents to combine active perception with external knowledge acquisition. Finally, we propose a multi-turn fatal-aware GRPO training algorithm that handles cascading tool failures by masking post-failure tokens while preserving useful pre-failure reasoning through one-sided advantage clamping. Built on this recipe, OpenSearch-VL delivers substantial performance gains, with over 10-point average improvements across seven benchmarks, and achieves results comparable to proprietary commercial models on several tasks. We will release all data, code, and models to support open research on multimodal deep search agents.

多模态搜索智能体强化学习开源

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。