arXiv:2410.02006cs.LGcs.AI2024-10中稿 · to TMLR, version o…

提出ANFR架构,用自适应归一化免处理联邦学习中的数据异构问题。

An Architecture Built for Federated Learning: Addressing Data Heterogeneity through Adaptive Normalization-Free Feature Recalibration

  • 融合权重标准化与通道注意力,动态调整特征分布应对数据异构。
  • 在多种数据集和聚合方法下,准确率提升3.2%~5.1%,优于现有基线。
  • 适用于全局与个性化联邦学习,且对差分隐私保护有良好兼容性。

联邦学习是一种去中心化协作训练范式,在保护数据所有权的同时提升模型性能与泛化能力。然而,客户端数据间的统计异构性会降低系统表现。为此,本文提出自适应无归一化特征重校准(ANFR),一种模型架构级方法,结合权重标准化与通道注意力机制,以应对联邦学习中的数据异构问题。ANFR利用权重标准化避免客户端统计量不匹配与平均不一致,增强异构环境下的鲁棒性;同时通过通道注意力生成可学习的缩放因子,抑制因异构导致的特征不一致性。实验表明,二者协同可显著提升模型性能,改善类别选择性与通道注意力权重分布。该方法兼容任意聚合策略,支持全局与个性化联邦学习,计算开销极低。在差分隐私训练下,ANFR实现隐私与效用的良好平衡,提供强隐私保障而不损失性能。通过将权重标准化与通道注意力集成至骨干网络,ANFR为统计异构挑战提供了一种新颖且通用的解决方案。大量实验证明,ANFR在不同聚合方式、数据集及异构条件下持续优于主流基线。代码已开源:https://github.com/siomvas/ANFR。

原文摘要 · Abstract (English)

Federated learning is a decentralized collaborative training paradigm preserving stakeholders' data ownership while improving performance and generalization. However, statistical heterogeneity among client datasets degrades system performance. To address this issue, we propose Adaptive Normalization-free Feature Recalibration (ANFR), a model architecture-level approach that combines weight standardization and channel attention to combat heterogeneous data in FL. ANFR leverages weight standardization to avoid mismatched client statistics and inconsistent averaging, ensuring robustness under heterogeneity, and channel attention to produce learnable scaling factors for feature maps, suppressing inconsistencies across clients due to heterogeneity. We demonstrate that combining these techniques boosts model performance beyond their individual contributions, by improving class selectivity and channel attention weight distribution. ANFR works with any aggregation method, supports both global and personalized FL, and adds minimal overhead. Furthermore, when training with differential privacy, ANFR achieves an appealing balance between privacy and utility, enabling strong privacy guarantees without sacrificing performance. By integrating weight standardization and channel attention in the backbone model, ANFR offers a novel and versatile approach to the challenge of statistical heterogeneity. Extensive experiments show ANFR consistently outperforms established baselines across various aggregation methods, datasets, and heterogeneity conditions. Code is provided at https://github.com/siomvas/ANFR.

联邦学习数据异构模型架构通道注意力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。