通过低秩更新提升联邦学习通信效率，同时增强模型稳定性。
Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularization
- 在客户端限制更新为低秩，降低通信开销
- 理论证明客户端梯度具更高秩结构，低秩近似更相似
- 适合大规模异构客户端场景，兼顾性能与效率
联邦学习在扩展至大量客户端时面临通信效率低下和性能下降的挑战。本文首次对联邦学习中的秩特性进行理论分析，发现客户端损失具有更高秩结构（即梯度跨越更高秩的海森子空间），且客户端梯度的低秩近似更具相似性。基于此，我们提出FedLoRU——一种通用的低秩更新框架：在客户端强制低秩更新，并累积生成更高秩的全局模型。该算法具备收敛性，收敛速率与FedAvg相当。此外，其变体可通过多层或分层低秩更新适应统计异构与模型异构环境。实验表明，FedLoRU在性能上可媲美全秩算法，在大规模异构客户端下仍保持鲁棒性。
原文摘要 · Abstract (English)
Federated Learning (FL) faces significant challenges related to communication efficiency and performance reduction when scaling to many clients. To address these issues, we explore the potential of using low-rank updates and provide the first theoretical study of rank properties in FL. Our theoretical analysis shows that a client's loss exhibits a higher-rank structure (i.e., gradients span higher-rank subspaces of the Hessian) compared to the server's loss, and that low-rank approximations of the clients' gradients have greater similarity. Based on this insight, we hypothesize that constraining client-side optimization to a low-rank subspace could provide an implicit regularization effect while reducing communication costs. Consequently, we propose FedLoRU, a general low-rank update framework for FL. Our framework enforces low-rank client-side updates and accumulates these updates to form a higher-rank model. We are able to establish convergence of the algorithm; the convergence rate matches FedAvg. Additionally, variants of FedLoRU can adapt to environments with statistical and model heterogeneity by employing multiple or hierarchical low-rank updates. Experimental results demonstrate that FedLoRU performs comparably to full-rank algorithms and exhibits robustness to heterogeneous and large numbers of clients.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。