arXiv:2410.23131cs.LGcs.DC2024-10NeurIPS被引 19

提出新算法,解决设备周期性参与下的联邦学习通信效率问题。

Federated Learning under Periodic Client Participation and Heterogeneous Data: A New Communication-Efficient Algorithm and Analysis

  • 设计放大版SCAFFOLD算法,应对客户端周期性参与
  • 理论证明通信轮数降至O(ε⁻²),比之前方法快得多
  • 适合大规模真实场景,尤其数据异构时表现更优

在联邦学习中,通常假设客户端始终可用,但实际用户设备难以满足。现有研究虽考虑周期或任意参与模式,但或需强假设、无法实现线性加速,或不适用于一般非凸设置。本文聚焦非凸优化,针对所有客户端在固定轮次内参与概率相等的参与模式(包含周期性参与),提出新算法Amplified SCAFFOLD。理论证明该算法可同时实现线性加速、减少通信轮数,并抵御数据异构。在周期性参与下,其在非凸随机设置中仅需$\mathcal{O}(ε^{-2})$轮通信即可找到$ε$-驻点;而此前方法需$\mathcal{O}(κ^2 ε^{-4})$轮,其中$κ$为数据异构度。分析基于对客户端参与与控制变量误差嵌套依赖的精细处理,获得更紧的理论保证。实验使用合成数据和真实数据($N=250$个客户端),验证了算法在周期性参与下的有效性。

原文摘要 · Abstract (English)

In federated learning, it is common to assume that clients are always available to participate in training, which may not be feasible with user devices in practice. Recent works analyze federated learning under more realistic participation patterns, such as cyclic client availability or arbitrary participation. However, all such works either require strong assumptions (e.g., all clients participate almost surely within a bounded window), do not achieve linear speedup and reduced communication rounds, or are not applicable in the general non-convex setting. In this work, we focus on nonconvex optimization and consider participation patterns in which the chance of participation over a fixed window of rounds is equal among all clients, which includes cyclic client availability as a special case. Under this setting, we propose a new algorithm, named Amplified SCAFFOLD, and prove that it achieves linear speedup, reduced communication, and resilience to data heterogeneity simultaneously. In particular, for cyclic participation, our algorithm is proved to enjoy $\mathcal{O}(ε^{-2})$ communication rounds to find an $ε$-stationary point in the non-convex stochastic setting. In contrast, the prior work under the same setting requires $\mathcal{O}(κ^2 ε^{-4})$ communication rounds, where $κ$ denotes the data heterogeneity. Therefore, our algorithm significantly reduces communication rounds due to better dependency in terms of $ε$ and $κ$. Our analysis relies on a fine-grained treatment of the nested dependence between client participation and errors in the control variates, which results in tighter guarantees than previous work. We also provide experimental results with (1) synthetic data and (2) real-world data with a large number of clients $(N = 250)$, demonstrating the effectiveness of our algorithm under periodic client participation.

联邦学习通信效率非凸优化周期参与

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。