提出新优化方法,在异步联邦学习中平衡模型延迟与更新频率。
Optimizing Asynchronous Federated Learning: A Delicate Trade-Off Between Model-Parameter Staleness and Update Frequency
- 通过随机建模分析异步联邦学习中的延迟与更新速度关系。
- 优化后准确率提升10%至30%,显著改善系统性能。
- 适合关注联邦学习效率与实用性的研究者与工程师。
同步联邦学习因慢速客户端(拖尾效应)导致扩展性差。现有异步算法如FedAsync和GeneralizedFedAsync通过允许客户端与服务器异步通信缓解该问题。本文基于随机建模与分析,深入理解异步联邦学习中并发度、路由概率等设计选择的影响,并利用此知识优化损失函数。相比以往研究,本文同时考虑客户端间服务速度的异质性与可变性,以及数据集的异质性。特别地,揭示了一个核心权衡:降低梯度估计误差需减少模型参数延迟,但提高系统吞吐量又需增加更新频率。本文两大贡献为:第一,证明离散版利特尔定律,导出相对延迟的闭式表达式,从而高效最小化每轮更新的平均损失(文献中标准指标),以Leconte等人的上界作为代理;第二,发现单纯优化该指标会严重降低系统速度,因此引入兼顾速度的新指标,并推导其可计算的上界进行数值优化。大量数值实验表明,该优化使准确率提升10%至30%。
原文摘要 · Abstract (English)
Synchronous federated learning (FL) scales poorly with the number of clients due to the straggler effect. Algorithms like FedAsync and GeneralizedFedAsync address this limitation by enabling asynchronous communication between clients and the central server. In this work, we rely on stochastic modeling and analysis to better understand the impact of design choices in asynchronous FL algorithms, such as the concurrency level and routing probabilities, and we leverage this knowledge to optimize loss. Compared to most existing studies, we account for the joint impact of heterogeneous and variable service speeds and heterogeneous datasets at the clients. We characterize in particular a fundamental trade-off for optimizing asynchronous FL: minimizing gradient estimation errors by avoiding model parameter staleness, while also speeding up the system by increasing the throughput of model updates. Our two main contributions can be summarized as follows. First, we prove a discrete variant of Little's law to derive a closed-form expression for relative delay, a metric that quantifies staleness. This allows us to efficiently minimize the average loss per model update, which has been the gold standard in literature to date, using the upper-bound of Leconte et al. as a proxy. Second, we observe that naively optimizing this metric drastically slows down the system by overemphasizing staleness at the expense of throughput. This motivates us to introduce an alternative metric that also accounts for speed, for which we derive a tractable upper-bound that can be minimized numerically. Extensive numerical results show these optimizations enhance accuracy by 10% to 30%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。