Laminar通过轨迹级异步设计,显著提升大模型强化学习训练效率。
Laminar: A Scalable Asynchronous RL Post-Training Framework
- 采用分层中继架构替代全局同步,实现细粒度权重更新
- 动态重打包机制将长尾轨迹集中处理,提升生成吞吐达5.48倍
- 适合大规模长期运行的LLM强化学习训练任务
大型语言模型(LLMs)的强化学习后训练正扩展至大规模集群并长时间运行,以提升模型推理能力。然而现有强化学习框架可扩展性受限,因轨迹生成存在极端长尾延迟分布,导致GPU严重闲置。当前异步系统虽尝试缓解,但依赖执行器与所有轨迹间的全局权重同步,形成僵化更新调度,难以适应轨迹生成延迟的高偏斜和动态变化,严重制约训练效率。我们提出Laminar,一种基于完全解耦架构的可扩展、鲁棒的强化学习后训练系统。首先,以中继工作节点构成分布式参数服务,取代全局更新,实现异步且细粒度的权重同步,使轨迹可随时拉取最新权重而不停滞执行器训练循环。其次,动态重打包机制将长尾轨迹集中于少数专用轨迹上,最大化生成吞吐。完全解耦设计还隔离故障,保障长期任务鲁棒性。在1024张GPU集群上的评估显示,相比最优系统,Laminar实现最高5.48倍训练吞吐提速,并缩短模型收敛时间。
原文摘要 · Abstract (English)
Reinforcement learning (RL) post-training for Large Language Models (LLMs) is now scaling to large clusters and running for extended durations to enhance model reasoning performance. However, the scalability of existing RL frameworks is limited, as extreme long-tail skewness in RL trajectory generation causes severe GPU underutilization. Current asynchronous RL systems attempt to mitigate this, but they rely on global weight synchronization between the actor and all rollouts, which creates a rigid model update schedule. This global synchronization is ill-suited for the highly skewed and evolving distribution of trajectory generation latency in RL training, crippling training efficiency. Our key insight is that efficient scaling requires breaking this lockstep through trajectory-level asynchrony, which generates and consumes each trajectory independently. We propose Laminar, a scalable and robust RL post-training system built on a fully decoupled architecture. First, we replace global updates with a tier of relay workers acting as a distributed parameter service. This enables asynchronous and fine-grained weight synchronization, allowing rollouts to pull the latest weight anytime without stalling the actor's training loop. Second, a dynamic repack mechanism consolidates long-tail trajectories onto a few dedicated rollouts, maximizing generation throughput. The fully decoupled design also isolates failures, ensuring robustness for long-running jobs. Our evaluation on a 1024-GPU cluster shows that Laminar achieves up to 5.48$\times$ training throughput speedup over state-of-the-art systems, while reducing model convergence time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。