让大模型强化学习训练更快，不改算法也能提速两倍以上。
Periodic Asynchrony: An On-Policy Approach for Accelerating LLM Reinforcement Learning
- 用周期性异步框架，把训练和推理分开执行，保持原算法不变。
- 在NPU上吞吐提升约2倍，GPU上最高提速3倍，精度几乎不变。
- 适合需要高效训练大模型的团队，尤其对算力敏感场景实用。
自GRPO算法问世以来，强化学习（RL）在大模型后训练中日益受到关注，但训练效率仍是关键挑战。主流RL框架将推理与训练部署在同一设备上，同步执行限制了并发性。本文重新审视推理与训练分离策略，提出一种周期性异步框架，将同步RL训练转化为异步生产者-消费者流水线。通过在每轮训练开始时同步模型权重，并用同一策略生成所有轨迹，该框架保持内在的在线策略特性——无需修改标准RL算法，避免了现有异步方法引入的离策略偏差。我们进一步设计统一的三模型架构和共享提示注意力机制，支持高效异步执行并减少冗余计算。在NPU平台上的实验显示，异步执行带来约2倍的吞吐提升，结合系统级优化后，端到端吞吐显著优于主流框架，GPU平台最高提速达3倍，充分验证了跨架构通用性，同时保持相近精度。该框架为可扩展的强化学习后训练提供了算法无关、实用且保持在线策略正确性的解决方案。代码已开源：https://github.com/janelu9/EasyLLM
原文摘要 · Abstract (English)
Since the introduction of the GRPO algorithm, reinforcement learning (RL) has attracted increasing attention for LLM post-training, yet training efficiency remains a critical challenge. In mainstream RL frameworks, inference and training are co-located on the same devices, and their synchronous execution prevents concurrent inference and training. In this work, we revisit the strategy of separating inference and training deployment, and propose a periodically asynchronous framework that transforms synchronous RL training into an asynchronous producer-consumer pipeline. By synchronising model weights at the beginning of each training iteration and generating all rollouts from the same policy, the proposed framework remains inherently on-policy -- without any modification to standard RL algorithms -- thereby avoiding the off-policy bias introduced by existing asynchronous approaches. We further introduce a unified tri-model architecture and a shared-prompt attention mechanism to support efficient asynchronous execution and reduce redundant computation. Experiments on NPU platforms show approximately 2x throughput improvement from asynchronous execution, with additional gains from system-level optimisations, substantially outperforming mainstream RL frameworks in end-to-end throughput, with speedups of up to 3x on GPU platforms, further confirming cross-architecture generalisability while maintaining comparable accuracy. The proposed framework thus offers a practical, algorithm-agnostic solution for scalable RL post-training without sacrificing on-policy correctness. Code available at: https://github.com/janelu9/EasyLLM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。