FedBKD通过生成数据实现双向知识蒸馏，同时提升全局与本地模型性能。
FedBKD: Distilled Federated Learning to Embrace Gerneralization and Personalization on Non-IID Data
- 用生成对抗网络生成合成数据，驱动全局与本地模型双向知识交互。
- 在4个基准上均达到当前最优，非独立同分布数据下表现稳定。
- 无需外部公共数据，避免隐私泄露，适合工业级隐私保护场景。
联邦学习（FL）是一种去中心化的协同机器学习方法，可解决工业实践中数据孤岛与隐私泄露问题。处理非独立同分布（non-IID）数据是其主要挑战：现有方法或追求强泛化全局模型，或定制个性化本地模型，难以兼顾两者。许多方案依赖公开数据集，增加泄露风险。为此，本文提出无数据知识蒸馏框架FedBKD。具体地，训练生成对抗网络（GAN）生成合成数据；训练过程中，本地模型作为判别器且参数冻结。利用合成数据在全局与本地模型间进行双向知识蒸馏，实现性能协同提升。在4个基准和多种non-IID设置下广泛实验表明，FedBKD在所有情况下均取得最先进性能。
原文摘要 · Abstract (English)
Federated learning (FL) is a decentralized collaborative machine learning (ML) technique. It provides a solution to the issues of isolated data islands and data privacy leakage in industrial ML practices. One major challenge in FL is handling the non-identical and independent distributed (non-IID) data. Current solutions either focus on constructing an all-powerful global model, or customizing personalized local models. Few of them can provide both a well-generalized global model and well-performed local models at the same time. Additionally, many FL solutions to the non-IID problem are benefited from introducing public datasets. However, this will also increase the risk of data leakage. To tackle the problems, we propose a novel data-free distillation framework, Federated Bidirectional Knowledge Distillation (FedBKD). Specifically, we train Generative Adversarial Networks (GAN) for synthetic data. During the GAN training, local models serve as discriminators and their parameters are frozen. The synthetic data is then used for bidirectional distillation between global and local models to achieve knowledge interactions so that performances for both sides are improved. We conduct extensive experiments on 4 benchmarks under different non-IID settings. The results show that FedBKD achieves SOTA performances in every case.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。