FedKDNAS让每个设备自选轻量模型,边训练边分享预测,提升效率和准确率。
Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search

- 客户端根据资源自主选模型,用蒸馏+监督联合训练
- 在非独立同分布数据下准确率最高提升15%,通信量减少44倍
- 适合算力不均、数据异构的边缘设备部署场景
联邦学习(FL)可在不集中数据的情况下实现协同建模,但实际应用需同时应对客户端数据的统计异构性(非独立同分布)、设备系统能力差异及通信效率问题。现有方法虽通过优化聚合、个性化或知识蒸馏缓解挑战,但普遍假设客户端使用固定架构,难以适应不同数据复杂度与硬件限制,导致准确率与效率权衡不佳。本文提出FedKDNAS——一种以蒸馏驱动的联邦框架,结合客户端神经架构搜索与服务器协调的知识蒸馏。每个客户端在精度-资源约束下自主选择轻量模型,本地训练时采用监督学习与知识蒸馏的混合目标,并仅共享在公共参考集上的预测结果。服务器对这些预测进行聚合与平滑,可选地融合教师模型,生成下一阶段稳定蒸馏目标。在六个数据集上对比六种代表性基线(FedAvg、Ditto、FedMD、FedDF、FedDistill、Local-KD)的实验表明,FedKDNAS始终表现更优:在非独立同分布条件下准确率最高提升15%,客户端CPU使用率降低约28%,通信开销减少高达44倍,且保持轻量化的logit通信。
原文摘要 · Abstract (English)
Federated Learning (FL) enables collaborative model training without centralizing data. However, real-world deployments must simultaneously address statistical heterogeneity across client data (non-IID), system heterogeneity in device capabilities, and communication efficiency. Existing FL approaches mitigate these challenges through improved aggregation, personalization, or knowledge distillation, but they almost universally assume a fixed client architecture, limiting adaptability to heterogeneous data complexity and hardware constraints. This architectural constraint often leads to suboptimal trade-offs between accuracy and efficiency in real-world FL systems. This work introduces FedKDNAS, a distillation-driven FL framework that combines client-side neural architecture selection with distillation of server-coordinated knowledge. Each client autonomously selects a lightweight model under accuracy-resource constraints. It then trains it locally using a hybrid objective combining supervised learning and knowledge distillation and shares only predictions on a public reference set. The server then aggregates and smooths these predictions, optionally combining them with a teacher model, to produce stable distillation targets for the next round. Extensive evaluation on six datasets against six representative FL baselines (FedAvg, Ditto, FedMD, FedDF, FedDistill, Local-KD) demonstrates that FedKDNAS consistently achieves superior Pareto efficiency, improving accuracy by up to 15\% under non-IID conditions, reducing client CPU usage by approximately 28\%, and decreasing communication overhead by up to 44 times while maintaining lightweight logit-based communication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。