提出轻量级边缘知识蒸馏方法,提升非独立同分布数据下模型协作效率。
Federated Distillation on Edge Devices: Efficient Client-Side Filtering for Non-IID Data
- 客户端用KMeans快速估算密度比,过滤异常数据
- 无需服务器端过滤,降低延迟且准确率接近独立同分布场景
- 适合资源受限边缘设备,可直接部署于实际系统
联邦知识蒸馏通过交换模型输出(软标签)而非完整参数,在保护隐私和减少通信方面优于传统联邦学习。然而,现有方法依赖计算开销大的统计密度比估计器来识别分布内代理数据,且需服务器端对模糊知识进行过滤,引入延迟。为此,我们提出高效、鲁棒的EdgeFD方法,降低客户端密度比估计复杂度,并取消服务器端过滤需求。EdgeFD采用基于KMeans的密度比估计器,有效在客户端过滤分布内与分布外代理数据,显著提升知识共享质量。我们在强非独立同分布、弱非独立同分布及独立同分布等多种实际场景下评估,无需服务器预训练教师模型即可实现蒸馏。实验表明,EdgeFD优于当前最优方法,在异构条件下持续达到接近独立同分布的准确率。基于KMeans的估计器计算开销极低,适用于资源受限的边缘设备,显著增强联邦蒸馏的可扩展性与实用性。代码已开源供复现。
原文摘要 · Abstract (English)
Federated distillation has emerged as a promising collaborative machine learning approach, offering enhanced privacy protection and reduced communication compared to traditional federated learning by exchanging model outputs (soft logits) rather than full model parameters. However, existing methods employ complex selective knowledge-sharing strategies that require clients to identify in-distribution proxy data through computationally expensive statistical density ratio estimators. Additionally, server-side filtering of ambiguous knowledge introduces latency to the process. To address these challenges, we propose a robust, resource-efficient EdgeFD method that reduces the complexity of the client-side density ratio estimation and removes the need for server-side filtering. EdgeFD introduces an efficient KMeans-based density ratio estimator for effectively filtering both in-distribution and out-of-distribution proxy data on clients, significantly improving the quality of knowledge sharing. We evaluate EdgeFD across diverse practical scenarios, including strong non-IID, weak non-IID, and IID data distributions on clients, without requiring a pre-trained teacher model on the server for knowledge distillation. Experimental results demonstrate that EdgeFD outperforms state-of-the-art methods, consistently achieving accuracy levels close to IID scenarios even under heterogeneous and challenging conditions. The significantly reduced computational overhead of the KMeans-based estimator is suitable for deployment on resource-constrained edge devices, thereby enhancing the scalability and real-world applicability of federated distillation. The code is available online for reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。