用推测解码加速强化学习训练中的推理生成，不损失输出质量。
Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding
- 在强化学习回放中引入推测解码，保持原模型输出分布不变。
- 8B模型下同步训练提速1.8倍，235B模型下端到端训练提速达2.5倍。
- 支持多种推测方法，适用于主流大模型训练流程。
前沿语言模型的强化学习后训练正面临自回归回放生成的瓶颈，回放加速成为核心系统挑战。现有高效方法多通过改变回放或优化策略提升吞吐量，如离策略执行、重放或低精度生成。本文研究推测解码作为无损加速原语，可保留目标模型的输出分布。我们在NeMo-RL中基于vLLM后端实现推测解码，支持同步与异步流水线，可在强化学习回放中启用推测。该优势适用于多种推测机制，如预训练的MTP头、小型外部草稿模型，甚至传统上用于强化学习后的Eagle3技术。这为当前最先进的推测解码提供了直接集成于强化学习训练的路径。在8B规模的推理后训练任务中，同步强化学习下推测解码使回放吞吐量提升1.8倍。通过高保真性能模拟器预测，在235B规模下结合推测解码与异步强化学习，端到端训练速度最高可提升2.5倍。
原文摘要 · Abstract (English)
RL post-training of frontier language models is increasingly bottlenecked by autoregressive rollout generation, making rollout acceleration a central systems challenge. Many existing efficiency methods improve throughput by changing the rollout or optimization regime, for example, through off-policy execution, replay, or lower-precision generation. We study speculative decoding as a lossless acceleration primitive for RL rollouts that preserves the target model's output distribution. We implement speculative decoding in NeMo-RL with a vLLM backend, supporting both synchronous and asynchronous pipelines and enabling speculation during RL rollouts. This benefit is realizable across speculation mechanisms, such as pretrained MTP heads, small external draft models or even techniques such as Eagle3, which are traditionally applied after RL phase. This yields a deployment path for state-of-the-art speculative decoding inside RL training. In a reasoning post-training workload at 8B scale under synchronous RL, speculative decoding improves rollout throughput by 1.8x. Using a high-fidelity performance simulator, we project that combining speculative decoding with asynchronous RL yields up to 2.5x end-to-end training speedup at 235B scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。