针对边缘联邦学习通信开销大问题,提出自适应压缩框架
Adaptive Heterogeneous Compression for Resource-Efficient Federated Knowledge Distillation

- 按客户端能力动态选择压缩策略,解决模型异构与资源不均难题
- 实验显示通信量降低40%以上,训练时间减少35%,精度保持竞争力
- 适合资源受限的边缘设备部署,尤其适用于异构模型场景
联邦学习(FL)实现隐私保护下的分布式模型训练,但面临模型架构异构和边缘网络通信资源有限的挑战。联邦知识蒸馏(FedKD)通过原型级参数聚合与跨异构模型的知识迁移缓解模型异构问题。然而,梯度传输仍带来显著通信开销,现有压缩方法通常对所有客户端采用统一策略,忽略其不同的模型特性与资源能力。为此,本文提出一种面向FedKD的异构压缩框架,使每个客户端从候选策略集中自主选择压缩方式。将压缩策略选择建模为非平稳随机多臂赌博机(MAB)问题,每条臂对应一种压缩策略。设计兼顾本地优化提升、全局知识对齐和执行时间的效率感知奖励函数。基于此,提出自适应异构压缩算法ASCEND,采用增强型指数移动平均(EMA)的ε-贪婪策略平衡探索与利用。在多个数据集上的实验表明,ASCEND能有效适配异构模型与资源环境,在降低通信开销和训练时间的同时,保持有竞争力的模型精度。
原文摘要 · Abstract (English)
Federated learning (FL) enables privacy-preserving distributed model training but faces challenges from heterogeneous model architectures and limited communication resources at the network edge. Federated knowledge distillation (FedKD) alleviates model heterogeneity by combining prototype-wise parameter aggregation and knowledge transfer across heterogeneous models. However, transmitting gradients still introduces considerable communication overhead, while existing compression approaches typically apply a uniform strategy across clients and ignore their diverse model characteristics and resource capacities. To address this issue, we propose a heterogeneous compression framework for FedKD that enables each client to select a compression strategy from a candidate strategy set. We formulate the compression strategy selection problem as a non-stationary stochastic multi-armed bandit (MAB), where each arm corresponds to a compression strategy. An efficiency-aware reward is designed by jointly considering local optimization improvement, global knowledge alignment, and execution time. Based on this formulation, we develop an Adaptive heterogeneouS Compression algorithm for fEderated kNowledge Distillation (ASCEND), which employs an exponential moving average (EMA)-enhanced $ε$-greedy policy to balance exploration and exploitation. Experimental results on multiple datasets demonstrate that ASCEND effectively adapts to heterogeneous model and resource settings, reducing communication overhead and training time while maintaining competitive model accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。