零数据训练智能体自主学习用工具，靠自我博弈进化。
Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Data
- 用自对弈强化学习，让生成器和求解器互挑难题、互学解法。
- 在多个工具使用基准上，性能比基础模型提升92.5%。
- 无需预设任务或人工标注，适合研究自主智能系统演化。
大型语言模型正成为能调用工具解决复杂任务的自主智能体的基础。强化学习是注入此类能力的常见方法，但通常依赖严格控制的训练环境，需精心设计的任务-解法配对与大量人工监督，这阻碍了向超智能系统的开放自演化。本文提出 Tool-R0 框架，可在零数据假设下，从零开始通过自对弈强化学习训练通用工具调用智能体。该框架由同一基础模型初始化的生成器与求解器组成，二者以互补奖励协同进化：一个在另一方能力边界提出针对性挑战任务，另一个则通过真实世界工具调用学习解决。此循环实现自我演进，无需预存任务或数据集。在多个工具使用基准上的评估显示，Tool-R0 相较基础模型相对提升92.5%，且在相同设置下超越全监督基线。本工作还通过分析协同进化、课程动态与扩展行为，为自对弈大模型智能体提供了实证洞察。
原文摘要 · Abstract (English)
Large language models (LLMs) are becoming the foundation for autonomous agents that can use tools to solve complex tasks. Reinforcement learning (RL) has emerged as a common approach for injecting such agentic capabilities, but typically under tightly controlled training setups. It often depends on carefully constructed task-solution pairs and substantial human supervision, which creates a fundamental obstacle to open-ended self-evolution toward superintelligent systems. In this paper, we propose Tool-R0 framework for training general purpose tool-calling agents from scratch with self-play RL, under a zero-data assumption. Initialized from the same base LLM, Tool-R0 co-evolves a Generator and a Solver with complementary rewards: one proposes targeted challenging tasks at the other's competence frontier and the other learns to solve them with real-world tool calls. This creates a self-evolving cycle that requires no pre-existing tasks or datasets. Evaluation on different tool-use benchmarks show that Tool-R0 yields 92.5 relative improvement over the base model and surpasses fully supervised tool-calling baselines under the same setting. Our work further provides empirical insights into self-play LLM agents by analyzing co-evolution, curriculum dynamics, and scaling behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。