arXiv:2511.14460cs.CL2025-11被引 31

提出统一框架,让智能体在多步交互中更高效学习。

Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learning

  • 以每步交互为基本单位建模强化学习
  • 支持多种信用分配方式,兼容不同优化策略
  • 适合构建复杂推理与工具使用的智能体系统

大型语言模型已从单轮文本生成演变为复杂智能体的基础。随着智能体承担更多元的推理、决策、工具使用和长程任务,强化学习在塑造其行为中的作用日益重要。传统将轨迹视为单一连续词元序列的方式,在多轮交互场景下变得不适用,导致上下文演化僵化且训练与回放间存在表征错配。本文提出 Agent-R1,一个围绕步骤级轨迹表示、灵活上下文管理及分层工作流接口构建的统一模块化框架。核心思想是将每个交互步骤视为强化学习的基本转移,同时保持优化层灵活性:一旦交互以步骤为单位建模,即可支持词元级、步骤级或其他兼容的信用分配设计。这一设计使框架能适配多种优化策略,而非绑定单一算法。整体架构提供了一个原则性强、可扩展、可复用的智能体强化学习基础。

原文摘要 · Abstract (English)

Large language models (LLMs) have rapidly evolved from single-turn text generators into the foundation of increasingly capable agents. As these agents take on more complex reasoning, decision making, tool use, and long-horizon tasks, reinforcement learning (RL) is becoming increasingly important for shaping their behavior. This shift is especially visible in agentic RL, where models must interact with tools and environments across multiple rounds rather than produce a single standalone response. In this regime, the usual view of a trajectory as one ever-growing token sequence becomes increasingly inadequate: it makes context evolution rigid and creates representation mismatches between rollout and training. This paper presents Agent-R1, a unified and modular framework for agentic RL built around step-level trajectory representation, flexible context management, and layered interfaces for workflows, environments and optimization. The key idea is to treat each interaction step as the basic reinforcement-learning transition, while keeping the optimization layer flexible: once the interaction is modeled at the step level, the framework can support token-level credit assignment, step-level credit assignment, or other compatible designs. These design choices make the framework compatible with a range of optimization strategies rather than tying it to a single algorithm. Together, these components provide a principled, extensible, and reusable substrate for agentic RL.

智能体强化学习框架设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。