分块上传参数，大幅降低大模型联邦学习通信开销。
FedBCD:Communication-Efficient Accelerated Block Coordinate Gradient Descent for Federated Learning
- 将模型参数分块，客户端只上传指定块，减少通信量。
- 理论证明通信复杂度比现有方法低1/N倍，收敛更快。
- 适合大规模模型联邦学习，尤其对资源受限设备友好。
尽管联邦学习近年受到广泛关注，但大型模型（如视觉变换器）在每轮通信中仍存在高昂开销。为降低通信复杂度，本文提出一种新型联邦块坐标梯度下降方法（FedBCGD），将模型参数划分为多个块，包括共享块，并允许客户端仅上传特定参数块，显著减少通信开销。进一步设计了具备客户端漂移控制与随机方差缩减的加速版本（FedBCGD+）。据我们所知，这是首次针对大规模深度模型参数块通信的研究。提供了算法收敛性分析，理论表明通信复杂度相比现有方法降低1/N倍（N为参数块数），且收敛速度更快。实验结果表明，所提算法优于当前最优方法。代码已开源：https://github.com/junkangLiu0/FedBCGD。
原文摘要 · Abstract (English)
Although Federated Learning has been widely studied in recent years, there are still high overhead expenses in each communication round for large-scale models such as Vision Transformer. To lower the communication complexity, we propose a novel Federated Block Coordinate Gradient Descent (FedBCGD) method for communication efficiency. The proposed method splits model parameters into several blocks, including a shared block and enables uploading a specific parameter block by each client, which can significantly reduce communication overhead. Moreover, we also develop an accelerated FedBCGD algorithm (called FedBCGD+) with client drift control and stochastic variance reduction. To the best of our knowledge, this paper is the first work on parameter block communication for training large-scale deep models. We also provide the convergence analysis for the proposed algorithms. Our theoretical results show that the communication complexities of our algorithms are a factor $1/N$ lower than those of existing methods, where $N$ is the number of parameter blocks, and they enjoy much faster convergence than their counterparts. Empirical results indicate the superiority of the proposed algorithms compared to state-of-the-art algorithms. The code is available at https://github.com/junkangLiu0/FedBCGD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。