针对异构设备的联邦学习,用可直接训练的压缩模型提升效率和准确率。
FedConv: A Learning-on-Model Paradigm for Heterogeneous Federated Clients
- 基于卷积压缩的模型学习范式,让不同设备运行定制化小模型。
- 平均准确率提升超35%,计算与通信开销分别降低33%和25%。
- 适合资源受限的移动端或边缘设备,尤其适合异构联邦场景。
联邦学习(FL)在不暴露客户端私有数据的前提下实现全局模型的协同训练。然而,在实际系统中,客户端(如边缘服务器、智能手机、可穿戴设备)通常具有不同的系统资源。传统联邦学习采用统一的大模型方案,将相同模型下发至各客户端,导致资源有限的客户端负载过重,而高算力客户端则无法充分参与。为此,我们提出 FedConv,一种面向客户端友好的联邦学习框架,通过提供异构定制子模型,显著降低资源受限客户端的计算与内存负担。FedConv 引入创新的“模型上学习”范式,利用卷积压缩方法学习异构子模型参数。与传统压缩方法不同,这些压缩模型可在客户端直接训练而无需解压。为聚合异构子模型,我们设计了转置卷积膨胀机制,将其还原为统一大小的大模型,同时保留客户端个性化信息。压缩与膨胀过程对客户端透明,由服务器基于小型公开数据集优化。在六个数据集上的大量实验表明,相比最先进方法,FedConv 在模型准确率上平均提升超过35%,计算与通信开销分别减少33%和25%。
原文摘要 · Abstract (English)
Federated Learning (FL) facilitates collaborative training of a shared global model without exposing clients' private data. In practical FL systems, clients (e.g., edge servers, smartphones, and wearables) typically have disparate system resources. Conventional FL, however, adopts a one-size-fits-all solution, where a homogeneous large global model is transmitted to and trained on each client, resulting in an overwhelming workload for less capable clients and starvation for other clients. To address this issue, we propose FedConv, a client-friendly FL framework, which minimizes the computation and memory burden on resource-constrained clients by providing heterogeneous customized sub-models. FedConv features a novel learning-on-model paradigm that learns the parameters of the heterogeneous sub-models via convolutional compression. Unlike traditional compression methods, the compressed models in FedConv can be directly trained on clients without decompression. To aggregate the heterogeneous sub-models, we propose transposed convolutional dilation to convert them back to large models with a unified size while retaining personalized information from clients. The compression and dilation processes, transparent to clients, are optimized on the server leveraging a small public dataset. Extensive experiments on six datasets demonstrate that FedConv outperforms state-of-the-art FL systems in terms of model accuracy (by more than 35% on average), computation and communication overhead (with 33% and 25% reduction, respectively).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。