降低分布式学习通信开销，支持大模型本地训练与随机梯度计算。
Communication-Efficient Stochastic Distributed Learning
- 基于分布式ADMM，每轮通信间执行多步本地训练，减少通信频率。
- 在非凸问题中收敛至驻点邻域，凸问题中收敛至最优解附近。
- 引入方差缩减实现精确收敛，本地训练还能加速整体收敛速度。
我们研究了在无向网络上的分布式学习问题，涵盖非凸与凸场景。提出一种新型算法，基于分布式交替方向乘子法（ADMM），以应对高通信成本和大规模数据的挑战。该设计通过允许代理在每轮通信之间执行多次本地训练步骤，并在本地计算中采用随机梯度，从而降低通信负担。我们证明，所提算法在非凸问题下收敛至驻点邻域，在凸问题下收敛至最优解邻域。进一步提出一种结合方差缩减的变体，实现精确收敛。理论与数值实验均表明，该算法在性能上优于现有方法，且本地训练可显著加速收敛。
原文摘要 · Abstract (English)
We address distributed learning problems, both nonconvex and convex, over undirected networks. In particular, we design a novel algorithm based on the distributed Alternating Direction Method of Multipliers (ADMM) to address the challenges of high communication costs, and large datasets. Our design tackles these challenges i) by enabling the agents to perform multiple local training steps between each round of communications; and ii) by allowing the agents to employ stochastic gradients while carrying out local computations. We show that the proposed algorithm converges to a neighborhood of a stationary point, for nonconvex problems, and of an optimal point, for convex problems. We also propose a variant of the algorithm to incorporate variance reduction thus achieving exact convergence. We show that the resulting algorithm indeed converges to a stationary (or optimal) point, and moreover that local training accelerates convergence. We thoroughly compare the proposed algorithms with the state of the art, both theoretically and through numerical results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。