用树形缓存加速语言模型强化学习，不牺牲准确性。
SRT: Accelerating Reinforcement Learning via Speculative Rollout with Tree-Structured Cache
- 用树结构缓存相同提示的生成结果，作为推测解码的草稿模型。
- 在多轮设置下，推理延迟降低，每令牌成本下降，最高速度提升2.08倍。
- 适合追求高效强化学习训练的开发者，尤其适用于PPO等标准算法。
我们提出基于树形缓存的推测推演（SRT），一种简单、无模型的方法，可在不牺牲分布正确性的前提下加速语言模型的在线策略强化学习。SRT利用相同提示在不同训练步中推演结果的统计相似性，将先前生成的延续内容存储于按提示组织的树形缓存中。生成时，当前策略使用该树作为草稿模型进行推测解码。为保持缓存更新并提升草稿质量，SRT从正在进行的推演中在线更新树结构，并在GPU空闲时段主动执行预推演生成。SRT可无缝集成至标准强化学习流程（如PPO、GRPO和DAPO）及多轮对话场景，在推演阶段实现高达2.08倍的墙钟时间加速，显著降低生成与步骤延迟，以及每令牌推理成本。
原文摘要 · Abstract (English)
We present Speculative Rollout with Tree-Structured Cache (SRT), a simple, model-free approach to accelerate on-policy reinforcement learning (RL) for language models without sacrificing distributional correctness. SRT exploits the empirical similarity of rollouts for the same prompt across training steps by storing previously generated continuations in a per-prompt tree-structured cache. During generation, the current policy uses this tree as the draft model for performing speculative decoding. To keep the cache fresh and improve draft model quality, SRT updates trees online from ongoing rollouts and proactively performs run-ahead generation during idle GPU bubbles. Integrated into standard RL pipelines (\textit{e.g.}, PPO, GRPO and DAPO) and multi-turn settings, SRT consistently reduces generation and step latency and lowers per-token inference cost, achieving up to 2.08x wall-clock time speedup during rollout.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。