让大模型强化学习在生成时就训练,大幅减少等待时间。
RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
- 将完整推理组流水线化,生成即开始训练。
- 比现有系统快30.7%~42.3%,训练空等时间降37%~76%。
- 适合需要高效推理训练的科学与逻辑任务场景。
大语言模型(LLM)在推理能力后训练中越来越多依赖可验证奖励的强化学习(RLVR),模型通过数学、逻辑和科学任务的真实反馈进行学习。为实现灵活资源分配并支持异构训练环境,现代RLVR系统采用解耦架构,将推理生成与策略训练分置于独立GPU池中。然而,现有同步式基于组的相对策略优化(GRPO)系统需完成全部推理才开始训练,导致训练器在推理期间闲置。异步流水线虽能重叠两阶段,但以使用过时数据为代价。为此,我们提出RolloutPipe,一种面向解耦式RLVR系统的后训练框架,将固定权重推理转为完整组流水线,使可训练组在后续组生成过程中即被送至训练器。该设计通过两种技术实现:完整组流水线(CGP)在每组生成完成后立即按先进先出原则发送至训练器;前沿组调度(FGD)在推理节点实施准入策略,优先接纳形成下一训练批次所需的前沿组,确保训练准备就绪的组更早、更稳定到达。该设计在推理完成前即启动训练,同时保持在线策略正确性。在四个推理与科学基准上对Qwen3-1.7B进行十二种推理设置评估,相比当前最先进的Slime系统,RolloutPipe将推理到训练结束的时间缩短30.7%~42.3%,并将训练器空等比例降低37%~76%。
原文摘要 · Abstract (English)
Large language model (LLM) post-training for reasoning increasingly relies on reinforcement learning with verifiable rewards (RLVR), where models learn from ground-truth feedback on mathematical, logical, and scientific tasks. To enable flexible resource allocation and support heterogeneous training setups, modern RLVR systems adopt disaggregated architectures that decouple rollout generation and policy training across independent GPU pools. However, existing synchronous on-policy GRPO (Group Relative Policy Optimization) RLVR systems finish an entire rollout before starting training, leaving the trainer GPU pool idle while rollout is still ongoing. Asynchronous RL pipelines overlap the two stages, but at the cost of training on stale data. To address these challenges, we propose RolloutPipe, a post-training framework for disaggregated RLVR systems, which turns the fixed-weight rollout into a complete-group pipeline where trainable groups move to the trainer while later groups are still being generated. RolloutPipe achieves this through two techniques including complete-group pipelining (CGP) and frontier-group dispatch (FGD). CGP dispatches each trainable complete group to the trainer FIFO as soon as group materialization finishes, and FGD is an admission policy on the Rollout node that first admits requests for the frontier groups needed to form the next training batch, so that trainer-ready groups arrive earlier and more steadily. The design starts training before the rollout completes while maintaining on-policy correctness. Evaluated on Qwen3-1.7B across four reasoning and science benchmarks and twelve rollout settings, RolloutPipe shortens the rollout-to-train-end time by 30.7%-42.3%, and lowers the trainer waiting ratio by 37%-76% compared to Slime, a state-of-the-art rollout and training system.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。