梯度裁剪让异步训练更抗慢节点，提升分布式学习稳定性。
Clipping Makes Distributed and Federated Asynchronous SGD Robust to Stragglers
- 用梯度裁剪消除延迟对收敛的负面影响
- 在重尾噪声下仍保证期望收敛与高概率收敛
- 适合大规模分布式深度学习训练场景
现代机器学习中，并行训练是提升规模的重要策略。异步随机梯度下降（ASGD）通过避免等待慢速工作节点，最大化硬件利用率。然而，使用固定步长时，由于更新延迟过大，ASGD的收敛性仍受慢节点影响。同时，实践中观察到深度学习异步训练中梯度裁剪能“稳定”训练。本文首次从理论上解释该现象：梯度裁剪可消除最优复杂度中对最大延迟的依赖。我们采用子威布尔（sub-Weibull）模型刻画梯度噪声，该模型推广了子高斯和子指数分布，更贴近深度学习中的重尾噪声特性。本文证明了期望收敛，并首次在异步优化中实现高概率收敛。
原文摘要 · Abstract (English)
In modern machine learning, parallelization of training is an important strategy for increasing scale. Asynchronous stochastic gradient descent (ASGD), which maximizes the utilization of available hardware by avoiding waiting for slow workers. However, with constant step sizes, the convergence of ASGD is nonetheless affected negatively by slow workers due to large delays in updates. At the same time, it has been empirically observed in asynchronous training of deep learning models that gradient clipping "stabilizes" training. In this work, we provide a theoretical justification for this behavior, as we show that clipping removes the dependence of the maximum delay in the oracle complexity. We employ a sub-Weibull model of gradient noise which generalizes sub-Gaussian and sub-exponential distributions to more heavy-tailed distributions, motivated by empirical observations in deep learning. We show convergence in expectation, and the first time in asynchronous optimization, convergence with high probability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。