利用等待时间生成预推演草案，让强化学习更快更准
BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learning
- 用快节点空闲时间预生成后续推演结果
- 减少50%解码步骤，吞吐提升1.8倍
- 无需预热，适配各种强化学习框架
强化学习（RL）已成为提升大语言模型性能的核心手段。然而，其推演阶段存在显著效率瓶颈，主要源于数据并行各节点间存在的长尾延迟问题，尤其在长上下文场景中，较快的GPU因等待慢节点而闲置。现有方法如部分推演或异步强化学习虽缓解了延迟，但牺牲了算法严格的同步性。本文提出BubbleSpec，一种在严格保持数学精确性的前提下加速推演的新框架。不同于消除延迟，BubbleSpec主动利用快速节点的空闲时段，预先生成后续步骤的推演结果，作为推测解码的草案。与依赖历史周期相似性和预热的先前方法不同，BubbleSpec对数据集大小无感，训练初期即可实现加速。大量实验表明，BubbleSpec将解码步骤减少50%，推演吞吐最高提升1.8倍。关键在于，该方法与多种强化学习框架和策略完全兼容，始终维持强化学习算法的严格同步性。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) has become a cornerstone for improving the performance of Large Language Models (LLMs). However, its rollout phase constitutes a significant efficiency bottleneck, mainly arising from the long-tail bubbles across data parallel ranks, particularly in long-context scenarios where faster GPUs remain idle while waiting for stragglers. Existing solutions, such as partial rollout or asynchronous RL, mitigate these bubbles by compromising the algorithm's strict synchronous nature. Instead, we propose BubbleSpec, a novel framework that accelerates RL rollouts while strictly keeping the mathematical exactness. Instead of attempting to eliminate bubbles, BubbleSpec exploits them. We exploit the idle time windows of faster ranks to pre-generate rollout results for subsequent steps, serving as drafts for speculative decoding. Unlike prior speculative methods that rely on historical epoch similarity and warm-ups, BubbleSpec is agnostic to dataset size and provides immediate acceleration from the onset of training. Extensive evaluations demonstrate that BubbleSpec reduces decoding steps by 50% and increases rollout throughput by up to 1.8x. Critically, BubbleSpec is seamlessly compatible with various RL frameworks and strategies as it sustains the strict synchronous property of RL algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。