针对联邦学习中固有类别差异问题，提出部分知识蒸馏方法提升弱类准确率。
Partial Knowledge Distillation for Alleviating the Inherent Inter-Class Discrepancy in Federated Learning
- 基于特定误分类触发部分知识蒸馏，针对性优化弱类性能。
- 弱类准确率提升10.7%，跨模型与数据划分下有效降低类别差距。
- 适用于长尾分布或类别不平衡场景下的联邦学习系统改进。
尽管已有大量研究致力于缓解联邦学习中的长尾类别分布问题，本文观察到一个有趣现象：即使在类别平衡的学习场景下，某些弱类仍持续存在。这些弱类不同于以往研究中的少数类，其根源在于数据本身，且对网络结构、学习范式和数据划分方式均保持稳定。在FashionMNIST和CIFAR-10数据集上，联邦学习的固有类别间准确率差距可超过36.9%。本文通过实证分析该现象成因，并提出一种部分知识蒸馏（PKD）方法，仅在特定弱类发生误分类时启动知识迁移。实验表明，弱类准确率平均提升10.7%，显著缓解了固有类别差异。
原文摘要 · Abstract (English)
Substantial efforts have been devoted to alleviating the impact of the long-tailed class distribution in federated learning. In this work, we observe an interesting phenomenon that certain weak classes consistently exist even for class-balanced learning. These weak classes, different from the minority classes in the previous works, are inherent to data and remain fairly consistent for various network structures, learning paradigms, and data partitioning methods. The inherent inter-class accuracy discrepancy can reach over 36.9% for federated learning on the FashionMNIST and CIFAR-10 datasets, even when the class distribution is balanced both globally and locally. In this study, we empirically analyze the potential reason for this phenomenon. Furthermore, a partial knowledge distillation (PKD) method is proposed to improve the model's classification accuracy for weak classes. In this approach, knowledge transfer is initiated upon the occurrence of specific misclassifications within certain weak classes. Experimental results show that the accuracy of weak classes can be improved by 10.7%, reducing the inherent inter-class discrepancy effectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。