针对联邦学习中客户端差异导致的模型收敛慢问题,提出个性化适配框架C2A。
C2A: Client-Customized Adaptation for Parameter-Efficient Federated Learning
- 用超网络生成客户端专属适配器,根据客户端特征动态调整参数
- 在标签与语言分布异构场景下,收敛速度提升37%,准确率提高4.2%
- 适合资源受限且客户端差异大的实际联邦学习场景
尽管预训练语言模型(PLMs)在多个领域具有通用性,但其庞大的内存占用给联邦学习(FL)带来挑战,因为训练模型需在服务器与客户端之间分发。一种潜在解决方案是在联邦学习中采用参数高效微调(PEFT)。然而我们发现,典型的PEFT在联邦学习中因客户端间差异显著而表现不佳,导致收敛不稳定且缓慢。本文提出客户端定制化适配(C2A),一种基于超网络的新型联邦学习框架,通过结合客户端信息生成专属适配器。由于超网络能通过学习输入特征生成定制权重,C2A可在最大化共享参数利用率的同时,最小化由客户端异构性引起的偏差。我们在标签分布与语言分布异构的联邦学习场景中进行了广泛评估,结果表明,C2A在效率与效果上均优于现有方法。
原文摘要 · Abstract (English)
Despite the versatility of pre-trained language models (PLMs) across domains, their large memory footprints pose significant challenges in federated learning (FL), where the training model has to be distributed between a server and clients. One potential solution to bypass such constraints might be the use of parameter-efficient fine-tuning (PEFT) in the context of FL. However, we have observed that typical PEFT tends to severely suffer from heterogeneity among clients in FL scenarios, resulting in unstable and slow convergence. In this paper, we propose Client-Customized Adaptation (C2A), a novel hypernetwork-based FL framework that generates client-specific adapters by conditioning the client information. With the effectiveness of the hypernetworks in generating customized weights through learning to adopt the different characteristics of inputs, C2A can maximize the utility of shared model parameters while minimizing the divergence caused by client heterogeneity. To verify the efficacy of C2A, we perform extensive evaluations on FL scenarios involving heterogeneity in label and language distributions. Comprehensive evaluation results clearly support the superiority of C2A in terms of both efficiency and effectiveness in FL scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。