解决分布式训练中的数据异构问题，加速模型收敛。
A Bias-Correction Decentralized Stochastic Gradient Algorithm with Momentum Acceleration
- 提出带动量的去偏差算法EDM，缓解数据异构带来的偏差。
- 非凸目标下亚线性收敛，弱于强凸的Polyak-Lojasiewicz条件下线性收敛。
- 理论分析更紧致，适用于深度学习中的动量型分布式算法。
分布式随机优化算法可并行处理大规模数据集，显著加速模型训练。然而，其效果常受分布式网络稀疏性和数据异构性制约。本文提出一种带动量的分布式随机梯度算法——精确扩散动量（Exact-Diffusion with Momentum, EDM），有效缓解数据异构带来的偏差，并引入深度学习中常用动量技术以提升收敛速度。理论分析表明，当应用于非凸目标函数时，EDM 算法以亚线性速率收敛至最优解邻域，该邻域半径与数据异构无关；在比强凸更弱的Polyak-Lojasiewicz条件下，可实现线性收敛。所采用的复杂分布式参数更新结构中处理动量的分析方法，给出了足够紧致的收敛上界，为其他动量型分布式算法的理论分析提供了新视角。
原文摘要 · Abstract (English)
Distributed stochastic optimization algorithms can simultaneously process large-scale datasets, significantly accelerating model training. However, their effectiveness is often hindered by the sparsity of distributed networks and data heterogeneity. In this paper, we propose a momentum-accelerated distributed stochastic gradient algorithm, termed Exact-Diffusion with Momentum (EDM), which mitigates the bias from data heterogeneity and incorporates momentum techniques commonly used in deep learning to enhance convergence rate. Our theoretical analysis demonstrates that the EDM algorithm converges sub-linearly to the neighborhood of the optimal solution, the radius of which is irrespective of data heterogeneity, when applied to non-convex objective functions; under the Polyak-Lojasiewicz condition, which is a weaker assumption than strong convexity, it converges linearly to the target region. Our analysis techniques employed to handle momentum in complex distributed parameter update structures yield a sufficiently tight convergence upper bound, offering a new perspective for the theoretical analysis of other momentum-based distributed algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。