让大模型用代码高效完成多步工具操作，提升真实任务成功率。
Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Use
- 通过生成可执行Python代码实现大模型的多步工具调用
- 在GAIA基准上比强基线高出约10%准确率，复杂任务提升更显著
- 适合需要可靠工具推理的智能体应用开发人员
大型语言模型在语言理解与推理方面表现出色，但在需要实时知识、精确操作或专业工具使用的真实任务中仍受限。为此，我们提出Tool-R1，一种强化学习框架，使大模型能够通过生成可执行的Python代码，实现通用、组合式及多步骤的工具使用。Tool-R1支持用户自定义工具与标准库集成，并在步骤间共享变量以构建连贯工作流。基于结果的奖励函数结合大模型答案判断与代码执行成功情况，指导策略优化。为提升训练效率，系统维护动态样本队列，缓存并重用高质量轨迹，降低昂贵在线采样的开销。在GAIA基准上的实验表明，Tool-R1显著提升了准确率与鲁棒性，相比强基线整体提升约10%，复杂多步任务中提升更明显。这些结果凸显了Tool-R1在真实应用场景中实现可靠、高效工具增强推理的潜力。代码将公开于https://github.com/YBYBZhang/Tool-R1。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated strong capabilities in language understanding and reasoning, yet they remain limited when tackling real-world tasks that require up-to-date knowledge, precise operations, or specialized tool use. To address this, we propose Tool-R1, a reinforcement learning framework that enables LLMs to perform general, compositional, and multi-step tool use by generating executable Python code. Tool-R1 supports integration of user-defined tools and standard libraries, with variable sharing across steps to construct coherent workflows. An outcome-based reward function, combining LLM-based answer judgment and code execution success, guides policy optimization. To improve training efficiency, we maintain a dynamic sample queue to cache and reuse high-quality trajectories, reducing the overhead of costly online sampling. Experiments on the GAIA benchmark show that Tool-R1 substantially improves both accuracy and robustness, achieving about 10\% gain over strong baselines, with larger improvements on complex multi-step tasks. These results highlight the potential of Tool-R1 for enabling reliable and efficient tool-augmented reasoning in real-world applications. Our code will be available at https://github.com/YBYBZhang/Tool-R1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。