提出自适应压缩方法，在联邦学习中提升通信效率与模型精度。
$γ$-FedHT: Stepsize-Aware Hard-Threshold Gradient Compression in Federated Learning
- 根据学习率动态调整梯度压缩阈值，降低计算开销至O(d)。
- 在非独立同分布数据上，相比Top-k提升最高7.42%准确率。
- 适合通信受限且数据异构的联邦学习场景，如边缘设备训练。
梯度压缩可有效缓解联邦学习中的通信瓶颈。当前先进的稀疏压缩方法如Top-k存在高计算复杂度，达$\mathcal{O}(d\log_2{k})$（$d$为参数量）。硬阈值压缩通过传输绝对值超过固定阈值的梯度元素，将复杂度降至$\mathcal{O}(d)$。然而在非独立同分布数据下，随着学习率$γ$递减，更新量减少导致硬阈值压缩比迅速下降至极低水平，引发严重精度损失。为此，我们提出$γ$-FedHT，一种学习率感知的低成本压缩方法，结合误差反馈以保证收敛性。由于传统联邦学习理论未涵盖误差反馈机制，我们建立了其基本收敛性分析。证明了在$μ$-强凸条件下，$γ$-FedHT具有$\mathcal{O}(\frac{1}{T})$的收敛速率（$T$为总迭代次数），在非凸情况下为$\mathcal{O}(\frac{1}{\sqrt{T}})$，与FedAVG一致。大量实验表明，在同等通信流量下，$γ$-FedHT在多种非独立同分布图像数据集上相比Top-k最高提升7.42%准确率。
原文摘要 · Abstract (English)
Gradient compression can effectively alleviate communication bottlenecks in Federated Learning (FL). Contemporary state-of-the-art sparse compressors, such as Top-$k$, exhibit high computational complexity, up to $\mathcal{O}(d\log_2{k})$, where $d$ is the number of model parameters. The hard-threshold compressor, which simply transmits elements with absolute values higher than a fixed threshold, is thus proposed to reduce the complexity to $\mathcal{O}(d)$. However, the hard-threshold compression causes accuracy degradation in FL, where the datasets are non-IID and the stepsize $γ$ is decreasing for model convergence. The decaying stepsize reduces the updates and causes the compression ratio of the hard-threshold compression to drop rapidly to an aggressive ratio. At or below this ratio, the model accuracy has been observed to degrade severely. To address this, we propose $γ$-FedHT, a stepsize-aware low-cost compressor with Error-Feedback to guarantee convergence. Given that the traditional theoretical framework of FL does not consider Error-Feedback, we introduce the fundamental conversation of Error-Feedback. We prove that $γ$-FedHT has the convergence rate of $\mathcal{O}(\frac{1}{T})$ ($T$ representing total training iterations) under $μ$-strongly convex cases and $\mathcal{O}(\frac{1}{\sqrt{T}})$ under non-convex cases, \textit{same as FedAVG}. Extensive experiments demonstrate that $γ$-FedHT improves accuracy by up to $7.42\%$ over Top-$k$ under equal communication traffic on various non-IID image datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。