调换更新顺序让联邦强化学习更稳定，提升收敛速度。
The Actor-Critic Update Order Matters for PPO in Federated Reinforcement Learning
- 先更新智能体再更新评价器，缓解客户端数据异构问题
- 理论证明在数据异构下仍能收敛，且性能不受影响
- 在5个场景中表现更优，尤其在交通模拟中效果显著
在联邦强化学习（FRL）中，将近端策略优化（PPO）应用于实际时，由于迭代间存在聚合步骤，其智能体与评价器的更新顺序会引发问题。当本地智能体基于本地评价器估计进行更新时，算法易受数据异构影响。传统更新顺序（先评后智）会导致各客户端梯度方向不一致，阻碍全局最优策略的收敛。为此，我们提出FedRAC，将更新顺序反转（先智后评），以消除不同客户端评价器之间的分歧。理论分析表明，在异构程度有界且策略评估准确的条件下，FedRAC的收敛边界不受数据异构影响。实验结果表明，该算法在五个场景中均获得更高累计奖励，并更快收敛，包括三个经典强化学习环境及使用SUMO交通模拟器的高异构自动驾驶场景。
原文摘要 · Abstract (English)
In the context of Federated Reinforcement Learning (FRL), applying Proximal Policy Optimization (PPO) faces challenges related to the update order of its actor and critic due to the aggregation step occurring between successive iterations. In particular, when local actors are updated based on local critic estimations, the algorithm becomes vulnerable to data heterogeneity. As a result, the conventional update order in PPO (critic first, then actor) may cause heterogeneous gradient directions among clients, hindering convergence to a globally optimal policy. To address this issue, we propose FedRAC, which reverses the update order (actor first, then critic) to eliminate the divergence of critics from different clients. Theoretical analysis shows that the convergence bound of FedRAC is immune to data heterogeneity under mild conditions, i.e., bounded level of heterogeneity and accurate policy evaluation. Empirical results indicate that the proposed algorithm obtains higher cumulative rewards and converges more rapidly in five experiments, including three classical RL environments and a highly heterogeneous autonomous driving scenario using the SUMO traffic simulator.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。