通过推测采样复用历史轨迹，加速大模型强化学习的推理阶段。
SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts
- 利用推测解码复用前序训练轮次的轨迹片段，减少重复生成。
- 在多个数学推理与泛化任务上，将推理时间缩短2至3倍。
- 无需修改主算法，可无缝集成到PPO等主流强化学习框架中。
大型语言模型越来越多地依赖具有可验证奖励的强化学习（RLVR）来获取可靠的思维链推理。然而，训练过程仍受计算成本高昂的采样阶段制约。现有加速方法如并行化、目标与数据驱动的修改以及重放缓冲区，或收益递减，或引入偏差，或忽略迭代间的冗余性。我们发现连续训练轮次的采样轨迹常存在大量重叠段，造成计算浪费。为此，提出SPEC-RL框架，将推测解码融入强化学习采样流程。该方法复用先前轨迹段作为推测前缀，通过‘草稿-验证’机制扩展，避免重复生成的同时保证策略一致性。在AIME24、MATH-500、OlympiadBench、MMLU-STEM等多个数学推理与泛化基准测试中，结果表明SPEC-RL可将采样时间降低2至3倍，且不损害策略质量。作为仅作用于采样阶段的增强方案，SPEC-RL可无缝集成至主流算法（如PPO、GRPO、DAPO），为大规模推理模型的强化学习提供通用且实用的加速路径。代码已开源：https://github.com/ShopeeLLM/Spec-RL
原文摘要 · Abstract (English)
Large Language Models (LLMs) increasingly rely on reinforcement learning with verifiable rewards (RLVR) to elicit reliable chain-of-thought reasoning. However, the training process remains bottlenecked by the computationally expensive rollout stage. Existing acceleration methods-such as parallelization, objective- and data-driven modifications, and replay buffers-either incur diminishing returns, introduce bias, or overlook redundancy across iterations. We identify that rollouts from consecutive training epochs frequently share a large portion of overlapping segments, wasting computation. To address this, we propose SPEC-RL, a novel framework that integrates SPECulative decoding with the RL rollout process. SPEC-RL reuses prior trajectory segments as speculative prefixes and extends them via a draft-and-verify mechanism, avoiding redundant generation while ensuring policy consistency. Experiments on diverse math reasoning and generalization benchmarks, including AIME24, MATH-500, OlympiadBench, MMLU-STEM, and others, demonstrate that SPEC-RL reduces rollout time by 2-3x without compromising policy quality. As a purely rollout-stage enhancement, SPEC-RL integrates seamlessly with mainstream algorithms (e.g., PPO, GRPO, DAPO), offering a general and practical path to scale RLVR for large reasoning models. Our code is available at https://github.com/ShopeeLLM/Spec-RL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。