针对个性化联邦学习中模型过拟合与知识丢失问题,提出分层自适应学习方法。
Optimizing Personalized Federated Learning through Adaptive Layer-Wise Learning
- 按层动态调整学习率,分角色优化本地模型训练
- 平均提升准确率5.40%(最高达14.29%)
- 适合数据非独立同分布场景的高效个性化建模
真实世界中的联邦学习常面临非独立同分布(non-IID)数据问题,导致精度下降和收敛缓慢。个性化联邦学习(pFL)通过为各客户端定制本地模型并采用加权聚合方法来应对。然而,现有方法难以在低计算开销下按需融合全局知识,且本地模型易过度拟合自身数据,丢失已有全局信息。我们提出FLAYER,一种面向pFL的分层自适应学习方法:根据神经网络各层的不同角色与学习能力,按需引入全局信息初始化本地模型,并在本地训练过程中动态调节各层学习率,实现个性化学习的同时保留全局知识。此外,为增强全局表征,FLAYER以分层方式选择性上传参数进行全局聚合。我们在计算机视觉与自然语言处理领域的四个代表性数据集上评估了FLAYER,相比六种先进pFL方法,其平均推理准确率提升5.40%(最高达14.29%)。
原文摘要 · Abstract (English)
Real-life deployment of federated Learning (FL) often faces non-IID data, which leads to poor accuracy and slow convergence. Personalized FL (pFL) tackles these issues by tailoring local models to individual data sources and using weighted aggregation methods for client-specific learning. However, existing pFL methods often fail to provide each local model with global knowledge on demand while maintaining low computational overhead. Additionally, local models tend to over-personalize their data during the training process, potentially dropping previously acquired global information. We propose FLAYER, a novel layer-wise learning method for pFL that optimizes local model personalization performance. FLAYER considers the different roles and learning abilities of neural network layers of individual local models. It incorporates global information for each local model as needed to initialize the local model cost-effectively. It then dynamically adjusts learning rates for each layer during local training, optimizing the personalized learning process for each local model while preserving global knowledge. Additionally, to enhance global representation in pFL, FLAYER selectively uploads parameters for global aggregation in a layer-wise manner. We evaluate FLAYER on four representative datasets in computer vision and natural language processing domains. Compared to six state-of-the-art pFL methods, FLAYER improves the inference accuracy, on average, by 5.40\% (up to 14.29\%).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。