arXiv:2411.03569cs.LGcs.AI2024-11中稿 · IEEE SMC 2024被引 4

用历史与全局模型指导本地训练,缓解个性化联邦学习中的遗忘问题。

Towards Personalized Federated Learning via Comprehensive Knowledge Distillation

  • 以全局和历史模型为教师,本地模型为学生,进行知识蒸馏。
  • 有效减少灾难性遗忘,提升个性化模型的性能表现。
  • 适合需要兼顾个性化与泛化能力的联邦学习场景。

联邦学习是一种保护数据隐私的分布式机器学习范式。然而,客户端间的数据异质性导致灾难性遗忘,即模型在学习新知识时迅速遗忘旧知识。为此,个性化联邦学习应运而生,为每个客户端定制个性化模型。但该机制过度强调个性化,可能损害模型的泛化能力。本文提出一种新型个性化联邦学习方法,利用全局模型和历史模型作为教师,本地模型作为学生,实现全面的知识蒸馏。历史模型代表上一轮客户端训练的本地模型,包含历史个性化知识;全局模型代表上一轮服务器聚合的模型,包含全局泛化知识。通过知识蒸馏,将全局泛化知识与历史个性化知识有效迁移至本地模型,从而缓解灾难性遗忘,提升个性化模型的整体性能。大量实验结果表明该方法具有显著优势。

原文摘要 · Abstract (English)

Federated learning is a distributed machine learning paradigm designed to protect data privacy. However, data heterogeneity across various clients results in catastrophic forgetting, where the model rapidly forgets previous knowledge while acquiring new knowledge. To address this challenge, personalized federated learning has emerged to customize a personalized model for each client. However, the inherent limitation of this mechanism is its excessive focus on personalization, potentially hindering the generalization of those models. In this paper, we present a novel personalized federated learning method that uses global and historical models as teachers and the local model as the student to facilitate comprehensive knowledge distillation. The historical model represents the local model from the last round of client training, containing historical personalized knowledge, while the global model represents the aggregated model from the last round of server aggregation, containing global generalized knowledge. By applying knowledge distillation, we effectively transfer global generalized knowledge and historical personalized knowledge to the local model, thus mitigating catastrophic forgetting and enhancing the general performance of personalized models. Extensive experimental results demonstrate the significant advantages of our method.

联邦学习知识蒸馏个性化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。