提出有序本地动量方法，解决异步分布式训练中的延迟问题。
Ordered Local Momentum for Asynchronous Distributed Learning under Arbitrary Delays
- 按全局迭代序聚合各节点本地动量，实现异步更新
- 理论证明在任意延迟下对非凸问题仍可收敛
- 实测优于同步及现有异步方法，适合异构计算环境
动量随机梯度下降（MSGD）因其加速收敛和提升泛化能力，在深度模型训练中扮演基础优化器角色。异步分布式学习对训练大规模深度模型至关重要，尤其在集群中各工作节点计算能力不均时。为减少通信频率，分布式学习普遍采用本地更新。然而，如何在存在本地更新的情况下实现异步分布式MSGD仍属空白。为此，我们提出一种新方法——有序本地动量（OrLoMo）。在OrLoMo中，每个工作节点独立运行本地MSGD，服务器则根据其全局迭代索引顺序聚合各节点的本地动量。据我们所知，OrLoMo是首个实现带本地更新的异步分布式MSGD的方法。我们在任意延迟条件下证明了OrLoMo对非凸问题的收敛性。实验表明，OrLoMo性能优于同步版本及其他异步方法。
原文摘要 · Abstract (English)
Momentum SGD (MSGD) serves as a foundational optimizer in training deep models due to momentum's key role in accelerating convergence and enhancing generalization. Meanwhile, asynchronous distributed learning is crucial for training large-scale deep models, especially when the computing capabilities of the workers in the cluster are heterogeneous. To reduce communication frequency, local updates are widely adopted in distributed learning. However, how to implement asynchronous distributed MSGD with local updates remains unexplored. To solve this problem, we propose a novel method, called \underline{or}dered \underline{lo}cal \underline{mo}mentum (OrLoMo), for asynchronous distributed learning. In OrLoMo, each worker runs MSGD locally. Then the local momentum from each worker will be aggregated by the server in order based on its global iteration index. To the best of our knowledge, OrLoMo is the first method to implement asynchronous distributed MSGD with local updates. We prove the convergence of OrLoMo for non-convex problems under arbitrary delays. Experiments validate that OrLoMo can outperform its synchronous counterpart and other asynchronous methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。