提出新算法，解决异步非凸优化中重尾噪声问题。
Optimal Asynchronous Stochastic Nonconvex Optimization under Heavy-Tailed Noise
- 设计带动量的异步归一化SGD算法应对重尾噪声。
- 在p阶矩有界的条件下达到最优时间复杂度，p∈(1,2]。
- 适合分布式训练中计算速度不均、噪声鲁棒性强的场景。
本文研究异步随机非凸优化在重尾梯度噪声及各工作节点计算时长任意异质条件下的问题。提出一种带动量的异步归一化随机梯度下降算法。分析表明，在假设梯度噪声的p阶中心矩有界且p∈(1,2]的条件下，该方法可实现最优时间复杂度。同时通过数值实验验证了所提方法的有效性。
原文摘要 · Abstract (English)
This paper considers the problem of asynchronous stochastic nonconvex optimization with heavy-tailed gradient noise and arbitrarily heterogeneous computation times across workers. We propose an asynchronous normalized stochastic gradient descent algorithm with momentum. The analysis show that our method achieves the optimal time complexity under the assumption of bounded $p$th-order central moment with $p\in(1,2]$. We also provide numerical experiments to show the effectiveness of proposed method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。