提出分块延迟梯度平均法，降低联邦学习通信开销
Delayed Random Partial Gradient Averaging for Federated Learning
- 客户端仅上传部分梯度，按更新率动态调整传输量
- 在非独立同分布数据上实现与全量传输相当的精度
- 支持计算通信并行，显著缩短系统运行时间
联邦学习（FL）是一种分布式机器学习范式，允许多个客户端在保护隐私的前提下协同训练共享模型。然而，真实世界中的FL系统常受两大通信瓶颈制约：(a) 尽管边缘设备算力提升支持部署大规模深度神经网络（DNN），但带宽有限导致大模型频繁传输困难；(b) 高延迟严重损害了FL性能。针对这些问题，我们提出了延迟随机部分梯度平均（DPGA）方法。在DPGA中，客户端仅向服务器上传本地模型的部分梯度，其传输规模由更新率决定，并随时间维度粗略初始化后逐步优化。此外，DPGA通过允许计算与通信并行，大幅减少系统运行时间。我们在非独立同分布的CIFAR-10/100数据集上进行了实验，验证了该方法的有效性。
原文摘要 · Abstract (English)
Federated learning (FL) is a distributed machine learning paradigm that enables multiple clients to train a shared model collaboratively while preserving privacy. However, the scaling of real-world FL systems is often limited by two communication bottlenecks:(a) while the increasing computing power of edge devices enables the deployment of large-scale Deep Neural Networks (DNNs), the limited bandwidth constraints frequent transmissions over large DNNs; and (b) high latency cost greatly degrades the performance of FL. In light of these bottlenecks, we propose a Delayed Random Partial Gradient Averaging (DPGA) to enhance FL. Under DPGA, clients only share partial local model gradients with the server. The size of the shared part in a local model is determined by the update rate, which is coarsely initialized and subsequently refined over the temporal dimension. Moreover, DPGA largely reduces the system run time by enabling computation in parallel with communication. We conduct experiments on non-IID CIFAR-10/100 to demonstrate the efficacy of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。