arXiv:2410.02845cs.LGcs.AI2024-10被引 5

根据梯度冲突程度动态决定哪些层个性化,提升联邦学习效果

Towards Layer-Wise Personalized Federated Learning: Adaptive Layer Disentanglement via Conflicting Gradients

  • 按层分析梯度冲突,冲突大则不参与全局聚合
  • 实验显示在多个数据集上优于现有方法
  • 适合数据差异大的联邦学习场景

在个性化联邦学习(pFL)中,高数据异质性会导致设备间梯度显著发散,尤其当不同用户梯度在聚合时形成钝角,会抵消学习进展,导致权重和梯度更新严重退化。为此,我们提出一种新方法——基于梯度分析的分层聚合联邦学习(FedLAG),利用层级梯度冲突概念。当不同客户端的层梯度夹角为锐角时,梯度方向一致,可共同学习客户端无关特征;当夹角为钝角时,层更关注客户端特定任务。因此,FedLAG依据层梯度冲突程度,将有冲突的层排除在全局聚合之外。理论分析表明,将该方法集成至其他pFL基线后,性能获得一定提升。大量实验验证,所提方法优于多个先进方法,且可轻松融入多种现有框架以进一步提升性能。

原文摘要 · Abstract (English)

In personalized Federated Learning (pFL), high data heterogeneity can cause significant gradient divergence across devices, adversely affecting the learning process. This divergence, especially when gradients from different users form an obtuse angle during aggregation, can negate progress, leading to severe weight and gradient update degradation. To address this issue, we introduce a new approach to pFL design, namely Federated Learning with Layer-wise Aggregation via Gradient Analysis (FedLAG), utilizing the concept of gradient conflict at the layer level. Specifically, when layer-wise gradients of different clients form acute angles, those gradients align in the same direction, enabling updates across different clients toward identifying client-invariant features. Conversely, when layer-wise gradient pairs make create obtuse angles, the layers tend to focus on client-specific tasks. In hindsights, FedLAG assigns layers for personalization based on the extent of layer-wise gradient conflicts. Specifically, layers with gradient conflicts are excluded from the global aggregation process. The theoretical evaluation demonstrates that when integrated into other pFL baselines, FedLAG enhances pFL performance by a certain margin. Therefore, our proposed method achieves superior convergence behavior compared with other baselines. Extensive experiments show that our FedLAG outperforms several state-of-the-art methods and can be easily incorporated with many existing methods to further enhance performance.

联邦学习个性化梯度分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。