通过分析生成长度分布，用历史数据加速强化学习的推理过程。
Beat the long tail: Distribution-Aware Speculative Decoding for RL Training
- 基于近期生成记录构建自适应草稿模型，动态优化生成效率。
- 针对长尾分布特性，对耗时长的轨迹分配更大草稿预算，提速最高50%。
- 适合需要高效训练大模型强化学习任务的研究者使用。
强化学习（RL）后训练已成为对齐大语言模型的关键步骤，但其效率受制于逐令牌生成的推演阶段。我们发现主要瓶颈在于推演长度的长尾分布：少数长序列占据大部分运行时间，同时存在可利用的历史推演数据，揭示了训练周期中提示层面的稳定模式。为此，我们提出分布感知的推测解码框架 DAS，可在不改变模型输出的前提下加速 RL 推演。DAS 结合两项核心思想：基于近期推演构建增量维护的后缀树实现非参数化自适应草稿器；以及长度感知的推测策略，为显著影响总完成时间的长序列分配更激进的草稿预算。该设计利用推演历史维持高接受率，同时平衡基础模型与令牌层级的成本。在数学和代码推理任务上的实验表明，DAS 最多可将推演时间减少 50%，且训练曲线完全一致，证明分布感知的推测解码能显著加速 RL 后训练而不牺牲学习质量。
原文摘要 · Abstract (English)
Reinforcement learning(RL) post-training has become essential for aligning large language models (LLMs), yet its efficiency is increasingly constrained by the rollout phase, where long trajectories are generated token by token. We identify a major bottleneck:the long-tail distribution of rollout lengths, where a small fraction of long generations dominates wall clock time and a complementary opportunity; the availability of historical rollouts that reveal stable prompt level patterns across training epochs. Motivated by these observations, we propose DAS, a Distribution Aware Speculative decoding framework that accelerates RL rollouts without altering model outputs. DAS integrates two key ideas: an adaptive, nonparametric drafter built from recent rollouts using an incrementally maintained suffix tree, and a length aware speculation policy that allocates more aggressive draft budgets to long trajectories that dominate makespan. This design exploits rollout history to sustain acceptance while balancing base and token level costs during decoding. Experiments on math and code reasoning tasks show that DAS reduces rollout time up to 50% while preserving identical training curves, demonstrating that distribution-aware speculative decoding can significantly accelerate RL post training without compromising learning quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。