构建可扩展的强化学习框架，让语言模型智能体更自主地完成任务。
AgentFly: Extensible and Scalable Reinforcement Learning for LM Agents
- 通过令牌级掩码适配传统强化学习，支持多轮交互。
- 实现异步工具调用与奖励计算，提升训练吞吐量。
- 提供插件式接口和预置工具，适合研究与开发人员快速搭建智能体。
语言模型（LM）智能体因其能通过与环境、工具和API交互自主完成任务而受到广泛关注。目前主流方法依赖提示工程或监督微调，而强化学习（RL）虽被探索用于提升推理与事实性能力，但语言模型智能体与强化学习的结合（Agent-RL）仍缺乏系统研究。为此，我们构建了AgentFly——一个可扩展、可延伸的Agent-RL框架，旨在为LM智能体赋能多种强化学习算法。框架通过令牌级掩码机制支持多轮交互，采用装饰器式接口定义工具与奖励函数，便于功能扩展与使用。为支持高吞吐训练，我们实现了工具调用与奖励计算的异步执行，并设计集中式资源管理系统以协调大规模环境。同时提供一系列预建工具与环境，验证了该框架在多任务场景中成功训练智能体的有效性。
原文摘要 · Abstract (English)
Language model (LM) agents have gained significant attention for their ability to autonomously complete tasks through interactions with environments, tools, and APIs. LM agents are primarily built with prompt engineering or supervised finetuning. At the same time, reinforcement learning (RL) has been explored to enhance LM's capabilities, such as reasoning and factuality. However, the combination of the LM agents and reinforcement learning (Agent-RL) remains underexplored and lacks systematic study. To this end, we built AgentFly, a scalable and extensible Agent-RL framework designed to empower LM agents with a variety of RL algorithms. Our framework supports multi-turn interactions by adapting traditional RL methods with token-level masking. It features a decorator-based interface for defining tools and reward functions, enabling seamless extension and ease of use. To support high-throughput training, we implement asynchronous execution of tool calls and reward computations, and design a centralized resource management system for scalable environment coordination. We also provide a suite of prebuilt tools and environments, demonstrating the framework's effectiveness through successful agent training across multiple tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。