不加梯度裁剪或归一化，普通SGD带动量仍可收敛于重尾噪声场景。
Vanilla SGD with Momentum Survives Heavy-Tailed Noise: Convergence Analysis without Gradient Clipping or Normalization
- 首次分析无梯度控制的SGD动量法在三类目标函数上的收敛性
- 证明其收敛速率低于裁剪/归一化版本，暴露原始方法局限
- 适合关注优化算法鲁棒性与理论边界的研究者
随机梯度下降（SGD）是现代优化的核心。尽管重尾噪声下通常通过梯度裁剪或归一化等改进方法应对，本文探讨一个更根本的问题：普通SGD（尤其是带动量版本）在重尾噪声下的表现如何？我们改进了现有对普通SGD的收敛分析，并首次提供了无梯度控制机制下普通SGD动量法在强凸、凸及非凸目标函数上的完整收敛分析。结果表明，其收敛速率劣于裁剪或归一化版本的最优速率，揭示了原始方法在重尾噪声下的固有局限。理论结论通过合成函数上的实验得到验证。
原文摘要 · Abstract (English)
Stochastic gradient descent (SGD) is a cornerstone of modern optimization. While its performance under heavy-tailed noise is often addressed through specialized modifications such as gradient clipping or normalization, we investigate a more fundamental question: how does vanilla SGD, particularly with momentum, perform in the presence of heavy-tailed noise? In this paper, we refine existing convergence results for vanilla SGD and, more importantly, provide the first comprehensive convergence analysis of vanilla SGD with momentum for strongly convex, convex, and nonconvex objectives, without employing any gradient control mechanisms. Our results demonstrate that the obtained convergence rates are inferior to the optimal rates achieved by clipped or normalized variants of SGD, thereby revealing inherent limitations of vanilla methods under heavy-tailed noise. The theoretical findings are supported by experiments on synthetic functions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。