通过自适应边缘的双知识蒸馏,提升异构联邦学习中原型聚合性能。
Dual-Distilled Heterogeneous Federated Learning with Adaptive Margins for Trainable Global Prototypes
- 引入双知识蒸馏与可训练原型,缓解原型边缘收缩问题。
- 在多个设置下平均提升1.13%,最高达34.13%准确率增长。
- 适合处理模型异构和非独立同分布数据的联邦学习场景。
异构联邦学习(HFL)因其能应对客户端间模型与数据异构性而受到广泛关注。基于原型的HFL方法通过共享类别代表性原型,有效缓解统计异构、模型异构及隐私挑战,推动了HFL研究进展。然而,传统加权平均聚合方式常导致全局原型决策边缘缩小,从而降低模型性能,尤其在模型异构与非独立同分布(non-IID)数据场景下表现更差。为此,本文提出FedProtoKD框架,在异构联邦学习中采用增强型双知识蒸馏机制,利用客户端的logits与原型特征表示提升系统性能。该框架通过基于对比学习的可训练服务器原型,结合类级自适应原型边缘,解决原型边缘收缩问题。同时,通过样本原型与类别代表原型的接近程度评估公共样本重要性,进一步优化学习效果。实验表明,FedProtoKD在多种设置下平均提升测试准确率1.13%,最高达34.13%,显著优于现有最先进HFL方法。
原文摘要 · Abstract (English)
Heterogeneous Federated Learning (HFL) has gained significant attention for its capacity to handle both model and data heterogeneity across clients. Prototype-based HFL methods emerge as a promising solution to address statistical and model heterogeneity as well as privacy challenges, paving the way for new advancements in HFL research. This method focuses on sharing class-representative prototypes among heterogeneous clients. However, aggregating these prototypes via standard weighted averaging often yields sub-optimal global knowledge. Specifically, the averaging approach induces a shrinking of the aggregated prototypes' decision margins, thereby degrading model performance in scenarios with model heterogeneity and non-IID data distributions. The propose FedProtoKD in a Heterogeneous Federated Learning setting, utilizing an enhanced dual-knowledge distillation mechanism to enhance system performance by leveraging clients' logits and prototype feature representations. The proposed framework aims to resolve the prototype margin-shrinking problem using a contrastive learning-based trainable server prototype by leveraging a class-wise adaptive prototype margin. Furthermore, the framework assess the importance of public samples using the closeness of the sample's prototype to its class representative prototypes, which enhances learning performance. FedProtoKD improved test accuracy by an average of 1.13% and up to 34.13% across various settings, significantly outperforming existing state-of-the-art HFL methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。