让大模型强化学习训练更快更省资源，支持异步执行
Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchrony
- 采用细粒度并行与推演-训练解耦设计，实现全异步训练
- 在相同硬件下，强化学习任务提速2.24倍，智能体任务提速2.72倍
- 适合需要高效训练大模型的团队，尤其关注资源利用率的场景
同步强化学习（RL）后训练已成为提升大语言模型多样能力的关键步骤。然而，现有加速系统仍存在资源利用率低、可扩展性差的问题。我们提出ROLL Flash，一种原生支持异步强化学习后训练的系统。该系统基于细粒度并行和推演-训练解耦两大设计原则，提供灵活编程接口，实现完全异步训练架构，并支持队列调度与环境级异步执行等高效推演机制。通过理论分析与大量实验，我们证明ROLL Flash显著提升资源利用率与可扩展性。在相同GPU预算下，其在RLVR任务上最高提速2.24倍，在智能体任务上达2.72倍。此外，我们实现了多种主流离策略算法，验证异步训练性能可媲美同步训练。
原文摘要 · Abstract (English)
Synchronous Reinforcement Learning (RL) post-training has emerged as a crucial step for enhancing Large Language Models (LLMs) with diverse capabilities. However, many systems designed to accelerate RL post-training still suffer from low resource utilization and limited scalability. We present ROLL Flash, a system that extends ROLL with native support for asynchronous RL post-training. ROLL Flash is built upon two core design principles: fine-grained parallelism and rollout-train decoupling. Guided by these principles, ROLL Flash provides flexible programming interfaces that enable a fully asynchronous training architecture and support efficient rollout mechanisms, including queue scheduling and environment-level asynchronous execution. Through comprehensive theoretical analysis and extensive experiments, we demonstrate that ROLL Flash significantly improves resource utilization and scalability over synchronous RL post-training. ROLL Flash achieves up to 2.24x speedup on RLVR tasks and 2.72x on agentic tasks, using the same GPU budget as synchronous baselines. Furthermore, we implement several popular off-policy algorithms and verify that asynchronous training can achieve performance on par with synchronous training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。