用沃瑟斯坦均值融合异构联邦强化学习模型，提升跨环境泛化能力。
Heterogeneous Federated Reinforcement Learning Using Wasserstein Barycenters
- 基于沃瑟斯坦均值聚合各智能体本地训练的网络参数
- 在变长杆小车任务中实现跨环境通用的全局DQN
- 适合解决非同质数据分布下的联邦强化学习问题
本文提出一种新型模型融合算法，利用沃瑟斯坦均值在分布式架构中训练全局深度神经网络（DNN）。将数据集均分给多个具有相同结构的智能体，每个智能体仅在其本地数据上训练；经过若干轮训练后，通过沃瑟斯坦均值聚合所有智能体的权重参数，形成名为FedWB的算法。进一步地，基于该机制开发了处理异构联邦强化学习（HFRL）的算法。实验以CartPole玩具问题为测试场景，通过改变杆长构造异构环境，在每个环境中训练一个深度Q网络（DQN）以控制小车，定期执行全局聚合步骤以增强局部模型的泛化能力，最终得到一个可在所有环境下运行的全局DQN。
原文摘要 · Abstract (English)
In this paper, we first propose a novel algorithm for model fusion that leverages Wasserstein barycenters in training a global Deep Neural Network (DNN) in a distributed architecture. To this end, we divide the dataset into equal parts that are fed to "agents" who have identical deep neural networks and train only over the dataset fed to them (known as the local dataset). After some training iterations, we perform an aggregation step where we combine the weight parameters of all neural networks using Wasserstein barycenters. These steps form the proposed algorithm referred to as FedWB. Moreover, we leverage the processes created in the first part of the paper to develop an algorithm to tackle Heterogeneous Federated Reinforcement Learning (HFRL). Our test experiment is the CartPole toy problem, where we vary the lengths of the poles to create heterogeneous environments. We train a deep Q-Network (DQN) in each environment to learn to control each cart, while occasionally performing a global aggregation step to generalize the local models; the end outcome is a global DQN that functions across all environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。