Seer通过学习上下文提升LLM强化学习的推理效率。
Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learning
- 根据相同提示的输出相似性动态分配任务，均衡负载。
- 相比现有系统，端到端吞吐提升2.04倍，长尾延迟降低72%-94%。
- 适合大规模在线强化学习场景，尤其关注响应速度与资源利用率。
强化学习（RL）已成为推动现代大语言模型（LLMs）发展的关键技术，但现有的同步强化学习系统存在严重性能瓶颈。生成阶段（rollout）主导了整体迭代时间，由于工作负载天然不均衡，导致显著的长尾延迟和资源利用不足。本文提出Seer，一种新型上下文学习的强化学习系统，其核心观察是：共享相同提示的请求在输出长度和响应模式上具有强相似性。基于此，Seer引入三项协同技术：(1) 分区式生成以实现动态负载均衡；(2) 上下文感知调度以缓解长尾请求延迟；(3) 自适应分组推测解码以加速生成。这些机制协同作用，显著降低长尾延迟并提升生成阶段的资源效率。在生产级强化学习工作负载上的评估表明，Seer相较于当前最优的同步强化学习系统，端到端生成吞吐最高提升2.04倍，长尾延迟降低72%-94%。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) has emerged as a critical technique for advancing modern Large Language Models (LLMs), yet existing synchronous RL systems face severe performance bottlenecks. The rollout phase, which dominates end-to-end iteration time, suffers from substantial long-tail latency and poor resource utilization due to inherent workload imbalance. We present Seer, a novel context learning RL system that addresses these challenges through a key observation: requests sharing the same prompt exhibit strong similarities in output lengths and response patterns. Leveraging this insight, Seer introduces three coordinated techniques: (1) divided rollout for dynamic load balancing, (2) context-aware scheduling to mitigate long-tail request delays, and (3) adaptive grouped speculative decoding to accelerate generation. These mechanisms work in concert to markedly reduce long-tail latency and improve resource efficiency during rollout. Evaluations on production-grade RL workloads demonstrate that Seer achieves up to 2.04$\times$ end-to-end rollout throughput improvement compared to the state-of-the-art synchronous RL systems, while notably reducing long-tail latency by 72-94%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。