在异构联邦学习中，用设备端知识蒸馏提升弱设备参与度。
Enabling Weak Client Participation via On-device Knowledge Distillation in Heterogeneous Federated Learning
- 设备端用小模型蒸馏本地标签数据，再通过无标签数据向大模型传知识。
- 相比顶尖方法，在非独立同分布数据下准确率显著提升。
- 适合资源差异大的边缘设备协同训练，尤其利好低配设备参与。
在线知识蒸馏（KD）近年来被用于联邦学习（FL）中训练大模型。现有研究多采用服务器端的软标签集成法，但通常假设边缘侧的无标签数据可集中于服务器，且该方法会个性化本地模型，导致软标签质量下降，尤其在数据高度非独立同分布（non-IID）时更为明显。为此，我们提出一种基于设备端知识蒸馏的异构联邦学习新方法。该方法首先利用小辅助模型从本地有标签数据中学习，随后由系统资源较强的客户端，通过设备端知识蒸馏，使用其无标签数据将知识传递给大模型。大量实验表明，所提方法有效利用了所有边缘设备的系统资源及无标签数据，相较于当前最优的基于KD的联邦学习方法，在准确率上实现显著提升。
原文摘要 · Abstract (English)
Online Knowledge Distillation (KD) is recently highlighted to train large models in Federated Learning (FL) environments. Many existing studies adopt the logit ensemble method to perform KD on the server side. However, they often assume that unlabeled data collected at the edge is centralized on the server. Moreover, the logit ensemble method personalizes local models, which can degrade the quality of soft targets, especially when data is highly non-IID. To address these critical limitations,we propose a novel on-device KD-based heterogeneous FL method. Our approach leverages a small auxiliary model to learn from labeled local data. Subsequently, a subset of clients with strong system resources transfers knowledge to a large model through on-device KD using their unlabeled data. Our extensive experiments demonstrate that our on-device KD-based heterogeneous FL method effectively utilizes the system resources of all edge devices as well as the unlabeled data, resulting in higher accuracy compared to SOTA KD-based FL methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。