提出新框架让大模型强化学习更高效，训练速度翻倍且性能不降。
How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
- 将GRPO分阶段大规模串行生成优化，大幅增加数据延迟但降低切换开销。
- 在五种模型、多个数学推理任务上，性能持平或超越标准GRPO，训练快约2倍。
- 创新使用松弛裁剪和负优势拦截，稳定老旧数据下的学习过程。
Group Relative Policy Optimization (GRPO) 是大语言模型强化学习中基于可验证奖励的关键进展，但通常在低延迟、近策略的环境中训练，系统开销较大。本文探讨：GRPO能多离策略？结果表明，GRPO类算法可容忍远超此前预期的数据延迟。为此提出Mu-GRPO框架，将训练划分为少量（如4个）大规模串行生成-优化阶段，实现高延迟采样，同时显著降低生成与优化间的切换开销。为稳定旧数据学习，引入松弛裁剪以保留有效梯度，结合负优势拦截机制，剔除负优势响应中的不稳定后续更新。在五种语言模型及多个数学推理基准上，Mu-GRPO性能与标准GRPO相当或更优，壁钟时间训练速度提升约2倍，显著改善了大模型强化学习的性能-效率权衡。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) has been a key driver of recent progress in reinforcement learning with verifiable rewards (RLVR) for large language models, but it is typically trained in a low-staleness, near-on-policy regime that incurs substantial system overhead. We ask a simple question: How off-policy can GRPO be? We show that GRPO-style algorithms can tolerate substantially larger rollout staleness than previously assumed, and propose Mu-GRPO, an RL training framework that organizes training into a small number (e.g., four) of large sequential generation-optimization stages. This design induces high rollout staleness while greatly reducing rollout-optimization switching overhead. To stabilize learning under stale data, Mu-GRPO combines relaxed clipping, which preserves useful stale-rollout gradients, with negative-advantage veto, which removes destabilizing post-trigger suffix updates in negative-advantage responses. Across five language models and multiple math reasoning benchmarks, Mu-GRPO matches or exceeds the performance of standard GRPO while achieving around 2x speedup in wall-clock training time, establishing a substantially improved performance-efficiency trade-off for LLM reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。