用强化学习让单智能体自主推理,提升深度研究能力。
SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents
- 通过纯合成数据的持续强化学习优化推理模型
- 在人类最后考试基准上达到28.7%正确率
- 适合需要自主搜索与推理的AI研究者
赋予大语言模型复杂的交错推理与工具使用能力,是智能体人工智能研究的关键方向,尤其在近期以“思考”为导向的模型发展背景下。此类能力对实现多项重要应用至关重要,例如深度研究(Deep Research),该任务需对大量信息源进行广泛检索与推理。本文聚焦于开发具备最小网页爬取和Python工具集成能力的原生自主单智能体模型。与多智能体系统中预设角色和静态流程不同,自主单智能体根据上下文动态决定下一步行动,无需人工指令。尽管已有工作提出基础或指令微调模型的训练方法,本文则专注于推理优化模型的持续强化学习,以进一步提升智能体能力,同时保持推理性能。为此,我们提出一种仅使用合成数据的简单强化学习方案,并应用于多个开源大模型。最佳版本SFR-DR-20B在Humanity's Last Exam基准上达到28.7%的得分。此外,我们还进行了关键分析实验,深入揭示方法的有效性。
原文摘要 · Abstract (English)
Equipping large language models (LLMs) with complex, interleaved reasoning and tool-use capabilities has become a key focus in agentic AI research, especially with recent advances in reasoning-oriented (``thinking'') models. Such capabilities are key to unlocking a number of important applications. One such application is Deep Research (DR), which requires extensive search and reasoning over many sources. Our work in this paper focuses on the development of native Autonomous Single-Agent models for DR featuring minimal web crawling and Python tool integration. Unlike multi-agent systems, where agents take up pre-defined roles and are told what to do at each step in a static workflow, an autonomous single-agent determines its next action dynamically based on context, without manual directive. While prior work has proposed training recipes for base or instruction-tuned LLMs, we focus on continual reinforcement learning (RL) of reasoning-optimized models to further enhance agentic skills while preserving reasoning ability. Towards this end, we propose a simple RL recipe with entirely synthetic data, which we apply to various open-source LLMs. Our best variant SFR-DR-20B achieves up to 28.7% on Humanity's Last Exam benchmark. In addition, we conduct key analysis experiments to provide more insights into our methodologies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。