FedPM通过预条件混合缓解联邦学习中参数漂移问题,提升模型收敛性。
FedPM: Federated Learning Using Second-order Optimization with Preconditioned Mixing of Local Parameters
- 在服务器端采用预条件参数混合,减少本地预条件器漂移。
- 单次本地更新下理论证明强凸目标具超线性收敛速度。
- 实验显示相比传统方法显著提升测试准确率,适合异构数据场景。
我们提出联邦预条件混合(FedPM),一种利用二阶优化的新型联邦学习方法。以往方法如LocalNewton、LTDA和FedSophia通过客户端迭代本地更新并在服务器端简单混合参数来引入二阶优化,但在异构数据设置下常因本地预条件器漂移而影响收敛。为此,我们将理想二阶更新(基于全局预条件梯度)分解为服务器端参数混合与客户端本地更新。结果,FedPM在服务器端引入了预条件参数混合,有效缓解了预条件器漂移问题。我们提供了理论分析,证明在单一本地更新条件下,强凸目标具有超线性收敛速率。大量实验表明,与采用简单混合的传统方法相比,FedPM在测试准确率上显著提升,充分释放了二阶优化潜力。
原文摘要 · Abstract (English)
We propose Federated Preconditioned Mixing (FedPM), a novel Federated Learning (FL) method that leverages second-order optimization. Prior methods--such as LocalNewton, LTDA, and FedSophia--have incorporated second-order optimization in FL by performing iterative local updates on clients and applying simple mixing of local parameters on the server. However, these methods often suffer from drift in local preconditioners, which significantly disrupts the convergence of parameter training, particularly in heterogeneous data settings. To overcome this issue, we refine the update rules by decomposing the ideal second-order update--computed using globally preconditioned global gradients--into parameter mixing on the server and local parameter updates on clients. As a result, our FedPM introduces preconditioned mixing of local parameters on the server, effectively mitigating drift in local preconditioners. We provide a theoretical convergence analysis demonstrating a superlinear rate for strongly convex objectives in scenarios involving a single local update. To demonstrate the practical benefits of FedPM, we conducted extensive experiments. The results showed significant improvements with FedPM in the test accuracy compared to conventional methods incorporating simple mixing, fully leveraging the potential of second-order optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。