研究了重尾噪声下弱凸优化的随机梯度方法收敛性,填补了非光滑非凸场景的理论空白。
Stochastic Weakly Convex Optimization Under Heavy-Tailed Noises
- 针对两类重尾噪声,分析了随机子梯度下降的高概率收敛性
- 在亚威布尔噪声下,原版方法收敛性不劣于光滑情况;在p阶矩有界噪声下,裁剪版本表现稳定
- 结果对深度学习中非光滑、非凸目标的优化提供了理论支持
越来越多研究关注在重尾梯度噪声下的随机一阶方法(SFOMs),这类噪声已在实际深度学习训练中被观测到。本文聚焦两种梯度噪声:亚威布尔噪声,以及具有有界p阶中心矩(p-BCM)的噪声(p∈(1,2])。后者更具挑战性,因当p∈(1,2)时方差无限。尽管在凸和光滑优化中已广泛研究了这两种噪声下的期望与高概率收敛性,但对于弱凸目标——包含所有Lipschitz连续凸函数与光滑函数——其在两种噪声下的收敛性理论仍不完整。本文研究了在亚威布尔噪声下原版随机子梯度下降(SsGD)的高概率收敛性,以及在p-BCM噪声下裁剪版SsGD的高概率与期望收敛性。结果显示,在亚威布尔噪声下,原版SsGD对失败概率与迭代次数的理论依赖性不劣于光滑情况;在p-BCM噪声下,非光滑性和非凸性不影响裁剪版对失败概率的依赖关系,但样本复杂度高于光滑优化的经典下界。
原文摘要 · Abstract (English)
An increasing number of studies have focused on stochastic first-order methods (SFOMs) under heavy-tailed gradient noises, which have been observed in the training of practical deep learning models. In this paper, we focus on two types of gradient noises: one is sub-Weibull noise, and the other is noise under the assumption that it has a bounded $p$-th central moment ($p$-BCM) with $p\in (1, 2]$. The latter is more challenging due to the occurrence of infinite variance when $p\in (1, 2)$. Under these two gradient noise assumptions, the in-expectation and high-probability convergence of SFOMs have been extensively studied in the contexts of convex optimization and standard smooth optimization. However, for weakly convex objectives-a class that includes all Lipschitz-continuous convex objectives and smooth objectives-our understanding of the in-expectation and high-probability convergence of SFOMs under these two types of noises remains incomplete. We investigate the high-probability convergence of the vanilla stochastic subgradient descent (SsGD) method under sub-Weibull noises, as well as the high-probability and in-expectation convergence of clipped SsGD under the $p$-BCM noises. Both analyses are conducted in the context of weakly convex optimization. For weakly convex objectives that may be non-convex and non-smooth, our results demonstrate that the theoretical dependence of vanilla SsGD on the failure probability and number of iterations under sub-Weibull noises does not degrade compared to the case of smooth objectives. Under $p$-BCM noises, our findings indicate that the non-smoothness and non-convexity of weakly convex objectives do not impact the theoretical dependence of clipped SGD on the failure probability relative to the smooth case; however, the sample complexity we derived is worse than a well-known lower bound for smooth optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。