arXiv:2501.00379cs.LGcs.IT2025-01被引 1

提出联邦丢弃的收敛理论,指导如何平衡通信与计算开销。

Federated Dropout: Convergence Analysis and Resource Allocation

  • 用泰勒展开分析丢弃率对梯度方差的影响,推导收敛性。
  • 丢弃率越高,收敛越慢;需权衡每轮延迟与总轮数。
  • 设计低复杂度算法联合优化丢弃率与带宽分配,适合边缘部署。

联邦丢弃是一种高效的边缘联邦学习技术,可缓解通信与计算瓶颈。每轮训练中,边缘设备仅需更新并传输由深度学习丢弃法生成的子模型,从而显著降低每轮延迟。然而,现有文献缺乏对联邦丢弃的理论收敛分析,尤其未量化丢弃率对收敛的影响。本文通过泰勒展开方法,数学证明梯度方差随缩放因子 $γ/(1-γ)$ 增大,其中 $γ∈[0,θ)$ 为丢弃率,$θ$ 为保证损失函数下降的最大丢弃率。基于该近似,给出了联邦丢弃的收敛分析:设备丢弃率越高,收敛速度越慢。这为在每轮延迟与总训练轮数间权衡提供了理论依据。此外,提出一种低复杂度算法,联合优化丢弃率与带宽分配,在满足每轮延迟和有限网络资源约束下,最小化所有轮次的损失函数。数值结果验证了所提算法的有效性。

原文摘要 · Abstract (English)

Federated Dropout is an efficient technique to overcome both communication and computation bottlenecks for deploying federated learning at the network edge. In each training round, an edge device only needs to update and transmit a sub-model, which is generated by the typical method of dropout in deep learning, and thus effectively reduces the per-round latency. \textcolor{blue}{However, the theoretical convergence analysis for Federated Dropout is still lacking in the literature, particularly regarding the quantitative influence of dropout rate on convergence}. To address this issue, by using the Taylor expansion method, we mathematically show that the gradient variance increases with a scaling factor of $γ/(1-γ)$, with $γ\in [0, θ)$ denoting the dropout rate and $θ$ being the maximum dropout rate ensuring the loss function reduction. Based on the above approximation, we provide the convergence analysis for Federated Dropout. Specifically, it is shown that a larger dropout rate of each device leads to a slower convergence rate. This provides a theoretical foundation for reducing the convergence latency by making a tradeoff between the per-round latency and the overall rounds till convergence. Moreover, a low-complexity algorithm is proposed to jointly optimize the dropout rate and the bandwidth allocation for minimizing the loss function in all rounds under a given per-round latency and limited network resources. Finally, numerical results are provided to verify the effectiveness of the proposed algorithm.

联邦学习丢弃法边缘计算优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。