提出新型联邦知识蒸馏框架,自动识别可靠样本提升学习效果。
FedeKD: Energy-Based Gating for Robust Federated Knowledge Distillation under Heterogeneous Settings

- 用能量门控机制动态评估样本可信度,实现细粒度知识迁移。
- 在六大数据集上显著降低异构环境下的负迁移,准确率提升1.8%~4.3%。
- 无需公开数据,适合医疗、金融等隐私敏感场景使用。
联邦学习在异构环境下运行,数据分布差异和模型设计不对称常导致负迁移。尽管联邦知识蒸馏(FKD)避免了直接共享模型参数,但现有方法通常依赖公共数据集或假设知识转移始终可靠,限制了实际鲁棒性。本文提出FedeKD,一种可靠性感知的联邦知识蒸馏框架,将样本级可信度估计作为显式知识迁移组件,不依赖额外公共数据。每个客户端维护一个高容量私有模型用于本地学习,以及一个轻量级共享代理模型用于跨客户端知识交换。训练过程中,代理模型在服务器端聚合形成全局代理,再用于指导私有模型更新。核心是基于能量的门控机制,将任务相关的私有-代理模型分歧转化为样本级信任权重,用于反向蒸馏。该机制实现样本级知识转移加权,使代理模型对可靠样本贡献更大,对不可靠样本降低权重。在六个真实世界数据集上的大量实验表明,FedeKD在异构设置下显著减少负迁移,同时保持强预测性能。
原文摘要 · Abstract (English)
Federated learning (FL) operates in heterogeneous environments, where variations in data distributions and asymmetric model design often result in negative transfer. While federated knowledge distillation (FKD) avoids direct model parameter sharing, existing methods typically rely on public datasets or assume that transferred knowledge is uniformly reliable, which limits their robustness in practice. This paper presents FedeKD, a reliability-aware FKD framework that makes sample-wise trust estimation an explicit component of knowledge transfer, without relying on additional public data. Each client maintains a high-capacity private model for local learning and a lightweight shared proxy model for cross-client knowledge exchange. During training, proxy models are aggregated on the server to form a global proxy, which is then used to guide updates of the private models. At the core of FedeKD is an energy-based gating mechanism that converts task-specific private-proxy disagreement into sample-wise trust weights for backward distillation. This mechanism enables sample-wise weighting of knowledge transfer, where the proxy model contributes more to reliable samples while down-weighting unreliable ones. Extensive experiments on six real-world datasets demonstrate that FedeKD significantly reduces negative transfer under heterogeneous settings while maintaining strong predictive performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。