RollArt通过拆分任务到不同硬件,提升大规模智能体强化学习训练效率。
RollArt: Disaggregated Multi-Task Agentic RL Training at Scale
- 按任务特性分配到最优硬件:推理、解码、环境运行分别用计算/带宽优化GPU和CPU集群。
- 轨迹级解耦与异步更新,使慢环境不阻塞其他流程,训练时间减少1.31–2.05倍。
- 支持超大规模模型训练,适合高并发、多任务的智能体强化学习场景。
智能体强化学习(Agentic RL)通过大语言模型与环境的多轮交互生成工作负载,包含计算密集型预填充、带宽敏感型解码、CPU-heavy环境执行及突发式奖励评估。现有系统要么将所有阶段集中部署于单一GPU集群,要么仅粗粒度解耦,忽视硬件异构性并引入显著同步开销。本文提出RollArt,一种在解耦基础设施上实现多任务智能体强化学习的系统。RollArt将各流水线阶段映射至最适硬件:预填充任务分配至计算优化GPU,解码任务至带宽优化GPU,环境运行部署于CPU集群。它在轨迹级别解耦回放过程,使生成、环境交互与奖励评分可独立进行,避免慢速或失败环境阻塞整体流程。此外,无状态奖励计算被卸载至无服务器架构,并通过受延迟约束的异步权重同步,实现训练与回放重叠。实验表明,RollArt有效提升训练吞吐量,在多种RL系统上实现1.31–2.05倍的训练时间缩短。我们还在阿里云超过3000个GPU的集群上,成功训练了一个百亿参数级MoE模型(用于Qoder产品),验证其稳定性与可扩展性。
原文摘要 · Abstract (English)
Agentic Reinforcement Learning (RL) trains LLMs through multi-turn interactions with environments, producing workloads that mix compute-bound prefill, bandwidth-bound decoding, CPU-heavy environment execution, and bursty reward evaluation. Existing systems either colocate all stages on a single GPU cluster or decouple them only at a coarse granularity, overlooking hardware heterogeneity and incurring substantial synchronization overhead across stages. We present ROLLART, a system for multi-task agentic RL on disaggregated infrastructure. ROLLART maps each pipeline stage to best-fit hardware, routing prefill-heavy tasks to compute-optimized GPUs, decode-heavy tasks to bandwidth-optimized GPUs, and environments to CPU clusters. It decouples rollout at the trajectory level, allowing generation, environment interaction, and reward scoring to proceed independently, so that slow or failed environments never block the others. ROLLART offloads stateless reward computation to serverless infrastructure and overlaps rollout with training via staleness-bounded asynchronous weight synchronization. Our results demonstrate that ROLLART effectively improves training throughput and achieves 1.31--2.05 \(\times\) training time reduction compared to various RL systems. We also evaluated ROLLART by training a hundreds-of-billions-parameter MoE model for Qoder product on an Alibaba cluster with above 3,000 GPUs, demonstrating its stability and scalability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。