提出GEPO算法，让分布式强化学习更稳定高效。
GEPO: Group Expectation Policy Optimization for Stable Heterogeneous Reinforcement Learning
- 用分组期望权重解耦参数更新与采样，降低延迟影响
- 1800秒延迟下性能仅降3%，最佳与最差结果差距缩小85%
- 适合资源异构、网络延迟高的分布式强化学习场景
随着单中心计算能力受限，去中心化训练成为必要。但传统强化学习方法因参数学习与采样高度耦合，难以适应分布式训练。为此，我们提出HeteroRL架构，通过解耦机制实现跨地理分布节点的稳定训练。核心是群组期望策略优化（GEPO），一种对网络延迟和算力异构鲁棒的异步强化学习算法。研究发现高延迟会显著增加KL散度，导致重要性权重方差上升，引发训练不稳定。GEPO通过群组期望加权，呈指数级降低重要性权重方差，并有理论保证。实验表明，GEPO在1800秒延迟下性能仅下降3%，最佳与最差结果差距相比GSPO减少85%（1.8 vs. 12.0），并取得最高得分，验证了其在异构分布式环境中的有效性。
原文摘要 · Abstract (English)
As single-center computing approaches power constraints, decentralized training becomes essential. However, traditional Reinforcement Learning (RL) methods, crucial for enhancing large model post-training, cannot adapt to decentralized distributed training due to the tight coupling between parameter learning and rollout sampling. For this, we propose HeteroRL, a heterogeneous RL architecture that decouples these processes, enabling stable training across geographically distributed nodes connected via the Internet. The core component is Group Expectation Policy Optimization (GEPO), an asynchronous RL algorithm robust to latency caused by network delays or heterogeneity in computational resources. Our study reveals that high latency significantly increases KL divergence, leading to higher variance of importance weights and training instability. GEPO mitigates this issue by using group expectation weighting to exponentially reduce the variance of importance weights, with theoretical guarantees. Experiments show GEPO achieves superior stability - only a 3% performance drop from online to 1800s latency-and reduces the best-to-last gap by 85% versus GSPO (1.8 vs. 12.0) while attaining the highest scores, highlighting its effectiveness in decentralized, resource-heterogeneous environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。