arXiv:2502.02727cs.LGcs.AI2025-02被引 1

解决联邦学习中自适应优化的梯度偏移问题,提升异构数据下的训练效果。

Gradient Correction in Federated Learning with Adaptive Optimization

  • 提出FAdamGC算法,在自适应优化中引入与动量结构匹配的梯度修正项。
  • 在非凸条件下证明收敛速率更优,且假设条件更宽松。
  • 实验证明在不同数据异构性下通信与计算成本更低,适合实际部署。

在联邦学习(FL)中,客户端间的数据异构性严重制约模型训练性能。为应对这一问题,近期出现了客户端漂移补偿方法,通过向本地更新注入修正项来改善效果。然而,现有方法仅适用于基于随机梯度下降(SGD)的训练,而现代联邦框架常采用自适应优化器(如Adam)以加速收敛。由于自适应优化器中一阶与二阶矩的复杂交互,直接套用针对SGD设计的修正项可能导致异构环境下性能下降。本文提出首个将漂移补偿集成至自适应联邦优化的算法FAdamGC,其核心思想是注入与自适应方法动量结构对齐的预估修正项。我们在非凸设置下提供了严格的收敛分析,表明FAdamGC的收敛速率更优,且假设条件更弱。实验结果表明,无论数据异构程度如何,FAdamGC均在总通信和计算成本上优于现有方法,验证了在自适应联邦优化中修正梯度信息的有效性。

原文摘要 · Abstract (English)

In federated learning (FL), model training performance is strongly impacted by data heterogeneity across clients. Client-drift compensation methods have recently emerged as a solution to this issue, introducing correction terms into local model updates. To date, these methods have only been considered under stochastic gradient descent (SGD)-based model training, while modern FL frameworks also employ adaptive optimizers (e.g., Adam) for improved convergence. However, due to the complex interplay between first and second moments found in most adaptive optimization methods, naively injecting correction terms can lead to performance degradation in heterogeneous settings. In this work, we propose {\tt FAdamGC}, the first algorithm to integrate drift compensation into adaptive federated optimization. The key idea of {\tt FAdamGC} is injecting a pre-estimation correction term that aligns with the moment structure of adaptive methods. We provide a rigorous convergence analysis of our algorithm under non-convex settings, showing that {\tt FAdamGC} results in better rate and milder assumptions than naively porting SGD-based correction algorithms into adaptive optimizers. Our experimental results demonstrate that {\tt FAdamGC} consistently outperform existing methods in total communication and computation cost across varying levels of data heterogeneity, showing the efficacy of correcting gradient information in federated adaptive optimization.

联邦学习自适应优化梯度修正

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。