FedSLoP通过低秩梯度投影,大幅降低联邦学习通信与内存开销。
FedSLoP: Memory-Efficient Federated Learning with Low-Rank Gradient Projection
- 用随机低秩子空间投影压缩梯度,减少通信和存储量
- 在异构数据上实现比FedAvg更低的通信量,准确率相当或更优
- 适合资源受限设备,尤其适用于内存有限的移动端联邦学习
联邦学习允许多个客户端协作训练模型而无需交换原始数据,但标准算法如FedAvg在异构、资源受限环境下存在收敛慢、通信与内存开销高的问题。本文提出FedSLoP,结合梯度的随机低秩子空间投影,降低传输与存储更新的维度,同时保持优化进度。理论上,在标准光滑性和有界方差假设下,建立了非凸收敛分析,证明其可保证以 $O(1/\\/sqrt{NT})$ 的速率收敛至一阶驻点。实验方面,在异构数据划分的联邦MNIST分类任务上进行了广泛测试,结果表明FedSLoP显著减少通信量与客户端内存占用,且准确率优于或相当於FedAvg及代表性稀疏或低秩基线方法。综合结果表明,如FedSLoP这类随机子空间动量方法为高效通信与内存的联邦学习提供了系统性有效方案。代码已开源:https://github.com/pkumelon/FedSLoP.git。
原文摘要 · Abstract (English)
Federated learning enables a population of clients to collaboratively train machine learning models without exchanging their raw data, but standard algorithms such as FedAvg suffer from slow convergence and high communication and memory costs in heterogeneous, resource-constrained environments. We introduce FedSLoP, a federated optimization algorithm that combines stochastic low-rank subspace projections of gradients, thereby reducing the dimension of communicated and stored updates while preserving optimization progress. On the theoretical side, we develop a detailed nonconvex convergence analysis under standard smoothness and bounded-variance assumptions, showing that FedSLoP is guaranteed to converge to a first-order stationary point at a rate of $O(1/\sqrt{NT})$. On the empirical side, we conduct extensive experiments on federated MNIST classification with heterogeneous data partitions, showing that FedSLoP substantially reduces communication volume and client-side memory while achieving competitive or better accuracy compared with FedAvg and representative sparse or low-rank baselines. Together, our results demonstrate that random subspace momentum methods such as FedSLoP provide a principled and effective approach to communication- and memory-efficient federated learning. Codes are available at: https://github.com/pkumelon/FedSLoP.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。