通过提前预取模型状态，显著加速联邦学习中的通信瓶颈。
FedFetch: Faster Federated Learning with Adaptive Downstream Prefetching
- 客户端提前多轮预取模型状态，减少等待时间。
- 在异构客户端下，训练时间缩短1.26倍，下载时间减少4.49倍。
- 适合资源受限、存在慢速客户端的分布式训练场景。
联邦学习（FL）是一种在边缘设备上利用用户数据进行大规模分布式模型训练的范式，由中心服务器协调。然而，部署中大量异构客户端导致服务器与客户端间通信成为瓶颈，尤其受慢速客户端拖累。现有技术如客户端采样和更新压缩虽各自有效，但在下游方向组合不佳：未选客户端因本地模型过时，需先同步状态。为此，我们提出FedFetch，一种自适应下游预取策略，使客户端提前多轮预取模型状态。实验表明，在异构客户端设置下，将FedFetch集成至高效通信的联邦学习方法中，可使端到端训练时间缩短1.26倍，下载时间减少4.49倍。代码已开源：https://github.com/DistributedML/FedFetch。
原文摘要 · Abstract (English)
Federated learning (FL) is a machine learning paradigm that facilitates massively distributed model training with end-user data on edge devices directed by a central server. However, the large number of heterogeneous clients in FL deployments leads to a communication bottleneck between the server and the clients. This bottleneck is made worse by straggling clients, any one of which will further slow down training. To tackle these challenges, researchers have proposed techniques like client sampling and update compression. These techniques work well in isolation but combine poorly in the downstream, server-to-client direction. This is because unselected clients have outdated local model states and need to synchronize these states with the server first. We introduce FedFetch, a strategy to mitigate the download time overhead caused by combining client sampling and compression techniques. FedFetch achieves this with an efficient prefetch schedule for clients to prefetch model states multiple rounds before a stated training round. We empirically show that adding FedFetch to communication efficient FL techniques reduces end-to-end training time by 1.26$\times$ and download time by 4.49$\times$ across compression techniques with heterogeneous client settings. Our implementation is available at https://github.com/DistributedML/FedFetch
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。