提出适用于异构联邦学习的高效分布式优化算法
Distributed optimization: designed for federated learning
- 基于增广拉格朗日法设计，支持多种通信拓扑
- 理论保证收敛性，实测在大规模异构数据下表现优异
- 可统一框架复现梯度下降等经典方法，适合研究者参考
联邦学习（FL）作为一种在隐私保护约束下实现跨组织数据协作的分布式机器学习框架，近年来受到广泛关注。本文提出一类基于增广拉格朗日技术的分布式优化算法，适用于中心化与去中心化两种联邦学习场景，并能适应多样的通信拓扑结构。通过引入近端松弛和二次逼近，对增广拉格朗日松弛进行推广，使该框架系统性地恢复了包括近端算法、经典梯度下降、随机梯度下降在内的多种经典无约束优化方法。在所提理论框架下，这些方法的收敛性质可自然推导。此外，我们设计了多种终止准则与参数更新机制以提升计算效率。数值实验表明，该算法在客户端存在显著统计异构性的大规模设置下仍表现出色。
原文摘要 · Abstract (English)
Federated learning (FL), as a distributed collaborative machine learning (ML) framework under privacy-preserving constraints, has garnered increasing research attention in cross-organizational data collaboration scenarios. This paper proposes a class of distributed optimization algorithms based on the augmented Lagrangian technique, designed to accommodate diverse communication topologies in both centralized and decentralized FL settings. Furthermore, we develop multiple termination criteria and parameter update mechanisms to enhance computational efficiency, accompanied by rigorous theoretical guarantees of convergence. By generalizing the augmented Lagrangian relaxation through the incorporation of proximal relaxation and quadratic approximation, our framework systematically recovers a broad of classical unconstrained optimization methods, including proximal algorithm, classic gradient descent, and stochastic gradient descent, among others. Notably, the convergence properties of these methods can be naturally derived within the proposed theoretical framework. Numerical experiments demonstrate that the proposed algorithm exhibits strong performance in large-scale settings with significant statistical heterogeneity across clients.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。