通过动态调度与缓存优化,显著提升长序列智能体强化学习的推理效率。
WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning

- 根据负载情况自动切换推测解码与缓存感知调度策略。
- 低负载下提速1.4倍,高负载下提速达1.6倍。
- 无需修改算法,适合大规模长上下文智能体训练场景。
长时序轨迹生成已成为智能体强化学习(RL)中的主要系统瓶颈。随着智能体在环境中多轮交互,轨迹迅速增长至数万令牌,使同步RL训练日益受限于轨迹生成。本文提出WAR,一种面向工作负载的轨迹生成系统,通过联合优化解码与调度,显著加速同步智能体强化学习。核心观察是:最优优化策略依赖运行时负载:(1)在低负载下,WAR采用无模型的推测解码(SuffixDecoding),复用先前完成轨迹的后缀模式作为未来轨迹的推测草稿。相比基于模型的草稿器,该方法无需额外模型,且避免了与轨迹生成的GPU竞争。(2)在高负载下,批处理解码饱和,推测加速空间有限,WAR将优化重点转向缓存感知调度。全局调度器基于缓存局部性、轨迹进度和服务器负载,在多个轨迹副本间分配请求,减少重复的键值缓存计算并缓解负载不均。结合解码层的后缀复用与系统层的调度优化,WAR在不同负载下均实现稳定吞吐提升,且无需更改底层RL算法。实验表明,其在低负载下将长上下文智能体轨迹生成吞吐提升1.4倍,在高负载下最高达1.6倍。结果证明,WAR有效消除了同步智能体强化学习中的主要轨迹瓶颈,为可扩展的长上下文智能体训练提供了实用路径。
原文摘要 · Abstract (English)
Long-horizon rollout generation has become the dominant systems bottleneck in agentic reinforcement learning (RL). As agents interact with environments over many turns, trajectories rapidly grow to tens of thousands of tokens, making synchronous RL training increasingly constrained by rollout. We propose WAR, a workload-aware rollout system that substantially accelerates synchronous agentic RL by jointly optimizing decoding and scheduling. WAR is built on a key observation: the optimal rollout optimization strategy depends on runtime load: (1) Under low load, WAR enables model-free speculative decoding with SuffixDecoding, which reuses suffix patterns from previously completed trajectories as speculative drafts for future rollouts. Unlike model-based drafters, SuffixDecoding introduces no additional draft model and avoids GPU contention with rollout generation. (2) Under high load, where saturated batched decoding leaves limited room for speculative speedup, WAR shifts the optimization focus to cache-aware scheduling. A global scheduler places requests across rollout replicas based on cache locality, trajectory progress and server load, reducing redundant KV-cache recomputation and mitigating load imbalance. By combining decoding-level suffix reuse with system-level rollout scheduling, WAR delivers robust throughput improvements across workload regimes without changing the underlying RL algorithm. WAR improves long-context agentic rollout throughput by 1.4x under low load and up to 1.6x under high load. These results show that WAR removes a major rollout bottleneck in synchronous agentic RL and provides a practical path toward scalable long-context agent training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。