用重参数优化器让小模型在联邦学习中逼近大模型性能
FedRepOpt: Gradient Re-parametrized Optimizers in Federated Learning
- 通过调整优化器梯度，让小模型模拟大模型训练效果
- 在VGG和Ghost模型上提升11.4%~16.7%准确率，收敛快57.4%
- 适合资源受限的边缘设备部署复杂模型
联邦学习（FL）是一种在边缘设备上分布式训练模型的隐私保护方法。然而，设备端模型受算力和内存限制，导致梯度更新能力受限。随着模型增大，边缘设备梯度更新频率下降，影响单轮训练效果，限制了大模型在边缘部署。为此，我们提出FedRepOpt，一种用于联邦学习的梯度重参数化优化器。该方法通过引入复杂模型获取的特定超参数，对优化器梯度进行重参数化，使简单本地模型获得接近复杂模型的性能。实验聚焦于VGG风格与Ghost风格模型，结果表明，使用FedRepOpt的模型相比RepGhost和RepVGG风格网络分别提升16.7%和11.4%性能，且收敛速度加快11.7%和57.4%。
原文摘要 · Abstract (English)
Federated Learning (FL) has emerged as a privacy-preserving method for training machine learning models in a distributed manner on edge devices. However, on-device models face inherent computational power and memory limitations, potentially resulting in constrained gradient updates. As the model's size increases, the frequency of gradient updates on edge devices decreases, ultimately leading to suboptimal training outcomes during any particular FL round. This limits the feasibility of deploying advanced and large-scale models on edge devices, hindering the potential for performance enhancements. To address this issue, we propose FedRepOpt, a gradient re-parameterized optimizer for FL. The gradient re-parameterized method allows training a simple local model with a similar performance as a complex model by modifying the optimizer's gradients according to a set of model-specific hyperparameters obtained from the complex models. In this work, we focus on VGG-style and Ghost-style models in the FL environment. Extensive experiments demonstrate that models using FedRepOpt obtain a significant boost in performance of 16.7% and 11.4% compared to the RepGhost-style and RepVGG-style networks, while also demonstrating a faster convergence time of 11.7% and 57.4% compared to their complex structure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。