通过主动部分回滚提升强化学习生成效率，解决长尾响应导致的算力浪费。
APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generation
- 超量请求+提前终止，回收未完成生成以减少等待时间。
- 平均提升22.5%的回滚吞吐量，最高达44%，加速收敛并提高准确率。
- 兼容多种框架与显卡，可直接部署于NVIDIA和AMD设备。
强化学习（RL）已成为推动大模型（如GPT-o系列、DeepSeek-R1、Kimi-K1.5、Grok 4、GLM-4.5）能力进化的核心。尽管已有众多RL框架提出，但训练仍高度依赖计算资源，其中回滚生成占总耗时超90%。尤其因响应长度呈长尾分布，少数长文本会阻塞整个批次，导致GPU空闲，严重制约扩展性。为此，本文提出主动部分回滚（APRIL），在回滚阶段超额预分配请求，一旦达到目标响应数即终止，并将未完成生成回收用于后续步骤。该策略避免丢弃任何回滚，显著降低GPU闲置时间。实验表明，APRIL在常见算法（GRPO、DAPO、GSPO）上平均提升22.5%回滚吞吐量（最高44%），加速收敛，任务最终准确率平均提升2.1%（最高8%）。该方法对框架与硬件无依赖，已集成至slime RL框架，支持NVIDIA与AMD GPU部署。本工作融合系统与算法视角，为强化学习训练效率提供新范式。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has become a cornerstone in advancing large-scale pre-trained language models (LLMs). Successive generations, including GPT-o series, DeepSeek-R1, Kimi-K1.5, Grok 4, and GLM-4.5, have relied on large-scale RL training to enhance reasoning and coding capabilities. To meet the community's growing RL needs, numerous RL frameworks have been proposed. However, RL training remains computationally expensive, with rollout generation accounting for more than 90% of total runtime. In addition, its efficiency is often constrained by the long-tail distribution of rollout response lengths, where a few lengthy responses stall entire batches, leaving GPUs idle and underutilized. As model and rollout sizes continue to grow, this bottleneck increasingly limits scalability. To address this challenge, we propose Active Partial Rollouts in Reinforcement Learning (APRIL), which mitigates long-tail inefficiency. In the rollout phase, APRIL over-provisions rollout requests, terminates once the target number of responses is reached, and recycles incomplete responses for continuation in future steps. This strategy ensures that no rollouts are discarded while substantially reducing GPU idle time. Experiments show that APRIL improves rollout throughput by 22.5% on average (at most 44%) across commonly used RL algorithms (GRPO, DAPO, GSPO), accelerates convergence, and achieves 2.1% on average(at most 8%) higher final accuracy across tasks. Moreover, APRIL is both framework and hardware agnostic, already integrated into the slime RL framework, and deployable on NVIDIA and AMD GPUs alike. Taken together, this work unifies system-level and algorithmic considerations in proposing APRIL, with the aim of advancing RL training efficiency and inspiring further optimizations in RL systems. Our codebase is available at https://github.com/RLsys-Foundation/APRIL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。