改进联邦学习中的优化方法，让训练更快更稳定。
FedMuon: Federated Learning with Bias-corrected LMO-based Optimization
- 用修正偏差的LMO优化器替代传统方法，提升局部训练效率。
- 实验表明，该方法在多个数据集上优于现有联邦学习算法。
- 适合关注联邦学习加速与收敛性研究的读者。
近期基于线性最小化算子（LMO）的优化方法Muon因其比Adam等自适应优化器更快的神经网络训练速度而受到关注。本文研究如何将Muon应用于联邦学习。我们首先发现，直接将Muon作为FedAvg的本地优化器时，由于LMO是带偏置的算子，无法收敛到驻点。为此，我们提出FedMuon以缓解此问题。进一步分析表明，近似求解LMO对收敛率的影响出人意料：无论牛顿-舒尔兹迭代次数多少，FedMuon均能收敛；且求解越精确，收敛越快。实验验证了该方法在多个数据集上超越当前最优联邦学习方法。
原文摘要 · Abstract (English)
Recently, a new optimization method based on the linear minimization oracle (LMO), called Muon, has been attracting increasing attention since it can train neural networks faster than existing adaptive optimization methods, such as Adam. In this paper, we study how Muon can be utilized in federated learning. We first show that straightforwardly using Muon as the local optimizer of FedAvg does not converge to the stationary point since the LMO is a biased operator. We then propose FedMuon which can mitigate this issue. We also analyze how solving the LMO approximately affects the convergence rate and find that, surprisingly, FedMuon can converge for any number of Newton-Schulz iterations, while it can converge faster as we solve the LMO more accurately. Through experiments, we demonstrated that FedMuon can outperform the state-of-the-art federated learning methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。