不用昂贵在线强化学习，也能训练出强大科研代理。
OffSeeker: Online Reinforcement Learning Is Not All You Need for Deep Research Agents
- 构建开源工具链，实现无需在线交互的离线训练。
- 生成66000个问答对与超2万条高质量训练数据。
- 80亿参数模型性能媲美300亿参数在线训练系统。
深度研究代理在处理长周期任务方面展现出巨大潜力。然而，当前顶尖性能通常依赖于昂贵的在线强化学习，因大量API调用导致成本高昂。离线训练虽更高效，但受限于高质量研究轨迹的稀缺。本文表明，仅靠在线强化学习并非打造强大研究代理的唯一路径。为此，我们推出一个完全开源的离线训练套件，核心包括DeepForge——一个可直接使用的任务合成框架，能无预处理生成大规模研究查询；以及包含66,000个问答对、33,000条SFT轨迹和21,000条DPO样本的精选数据集。基于这些资源，我们训练了完全离线的OffSeeker（8B）模型。在六个基准上的广泛评估显示，OffSeeker不仅在同类规模代理中表现领先，且在性能上仍可与采用大规模在线强化学习训练的300亿参数系统相竞争。
原文摘要 · Abstract (English)
Deep research agents have shown remarkable potential in handling long-horizon tasks. However, state-of-the-art performance typically relies on online reinforcement learning (RL), which is financially expensive due to extensive API calls. While offline training offers a more efficient alternative, its progress is hindered by the scarcity of high-quality research trajectories. In this paper, we demonstrate that expensive online reinforcement learning is not all you need to build powerful research agents. To bridge this gap, we introduce a fully open-source suite designed for effective offline training. Our core contributions include DeepForge, a ready-to-use task synthesis framework that generates large-scale research queries without heavy preprocessing; and a curated collection of 66k QA pairs, 33k SFT trajectories, and 21k DPO pairs. Leveraging these resources, we train OffSeeker (8B), a model developed entirely offline. Extensive evaluations across six benchmarks show that OffSeeker not only leads among similar-sized agents but also remains competitive with 30B-parameter systems trained via heavy online RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。