解决大模型智能体强化学习的长上下文训练瓶颈
EARL: Efficient Agentic Reinforcement Learning Systems for Large Language Models
- 动态调整模型与训练并行策略，适配不同阶段的序列长度
- 通过布局感知的数据分发机制，降低跨设备通信开销
- 显著提升吞吐量，支持稳定的大规模智能体训练
强化学习已成为大语言模型后训练的关键组件，而智能体强化学习通过多轮交互和工具使用扩展了该范式。然而，系统规模扩大时面临两大实际瓶颈：(1) 训练过程中上下文长度快速增长，导致内存占用和延迟上升，引发内存溢出（OOM）；(2) 中间张量随上下文增长累积，使跨设备数据传输成为主要性能瓶颈。我们提出EARL，一种高效的智能体强化学习系统。EARL设计了一个并行性选择器，根据序列长度和系统负载动态调整各训练阶段的模型与训练并行度；同时引入数据调度器，实现布局感知、去中心化的中间数据批次交换。二者协同提升了训练吞吐量，减少了长上下文失败率，使智能体大模型在不依赖上下文长度硬限制或惩罚机制的情况下实现稳定大规模训练。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has become a pivotal component of large language model (LLM) post-training, and agentic RL extends this paradigm to operate as agents through multi-turn interaction and tool use. Scaling such systems exposes two practical bottlenecks: (1) context length grows rapidly during training, inflating memory usage and latency, and triggering out-of-memory (OOM) failures; and (2) intermediate tensors accumulate with context length, making cross-device data movement a major system bottleneck. We present EARL, a scalable system for efficient agentic RL. EARL designs a parallelism selector that dynamically adapts model and training parallelism across RL stages based on sequence length and system load, and a data dispatcher that performs layout-aware, decentralized exchange of intermediate data batches. Together, these components increase throughput, reduce long-context failures, and enable stable large-scale training of agentic LLMs without relying on hard limits or penalties of context length.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。