针对少部分客户端参与的联邦学习难题，提出知识蒸馏新方法提升模型性能。
Tackling Data Heterogeneity in Federated Learning through Knowledge Distillation with Inequitable Aggregation
- 用不均等加权聚合所有客户端构建教师模型，融合参与频率、次数和数据量。
- 在严重数据异构下，减少训练轮次仍能实现更高准确率，提升显著。
- 适合大规模异构联邦学习场景，尤其客户端参与不均衡时使用。
联邦学习旨在分布式环境下训练全局模型，使其性能接近集中式训练。然而，客户端标签偏移、数据量偏移及其他异构问题会严重降低模型表现。现有方法常忽略大规模客户端中仅有少数参与训练的情形，而实验表明该场景更具挑战性。为此，我们提出一种面向此场景的知识蒸馏与不均等聚合策略（KDIA），可有效利用所有客户端的知识。在KDIA中，学生模型为参与客户端的平均聚合，教师模型则基于参与间隔、参与次数和数据量比例进行加权聚合。本地训练时执行自知识蒸馏，并在服务器端训练生成器，生成近似独立同分布（IID）的数据特征用于辅助训练。我们在CIFAR-10/100/CINIC-10数据集上，于多种异构设置下进行了大量实验。结果表明，KDIA能在更少训练轮次下取得更高准确率，且在严重异构条件下优势更明显。
原文摘要 · Abstract (English)
Federated learning aims to train a global model in a distributed environment that is close to the performance of centralized training. However, issues such as client label skew, data quantity skew, and other heterogeneity problems severely degrade the model's performance. Most existing methods overlook the scenario where only a small portion of clients participate in training within a large-scale client setting, whereas our experiments show that this scenario presents a more challenging federated learning task. Therefore, we propose a Knowledge Distillation with teacher-student Inequitable Aggregation (KDIA) strategy tailored to address the federated learning setting mentioned above, which can effectively leverage knowledge from all clients. In KDIA, the student model is the average aggregation of the participating clients, while the teacher model is formed by a weighted aggregation of all clients based on three frequencies: participation intervals, participation counts, and data volume proportions. During local training, self-knowledge distillation is performed. Additionally, we utilize a generator trained on the server to generate approximately independent and identically distributed (IID) data features locally for auxiliary training. We conduct extensive experiments on the CIFAR-10/100/CINIC-10 datasets and various heterogeneous settings to evaluate KDIA. The results show that KDIA can achieve better accuracy with fewer rounds of training, and the improvement is more significant under severe heterogeneity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。