解释为何归一化在重尾噪声下优于截断,理论证明其收敛性更优。
Why is Normalization Preferred? A Worst-Case Complexity Theory for Stochastically Preconditioned SGD under Heavy-Tailed Noise
- 用归一化方法可保证在重尾噪声下稳定收敛
- 归一化在已知/未知参数时分别达最优收敛率
- 揭示大规模训练中归一化被偏好的理论原因
我们为随机预处理随机梯度下降(SPSGD)及其加速版本在重尾噪声下的最坏情况复杂度建立了理论框架,该设置包含Adam、RMSProp和Shampoo等常用自适应方法。假设随机梯度噪声具有有限p阶矩(p∈(1,2]),衡量经T次迭代后的收敛性。尽管截断与归一化都是应对重尾噪声的稳定手段,但在随机预处理场景下二者最坏情况性质存在根本差异。我们证明:归一化在已知问题参数时可实现$\ ext{O}(T^{-\frac{p-1}{3p-2}})$的收敛率,在未知参数时为$\ ext{O}(T^{-\frac{p-1}{2p}})$,分别匹配归一化SGD的最优率;而截断可能因预处理器与梯度估计间的统计依赖而在最坏情况下无法收敛。为此,我们推导出一种新型向量值Burkholder型不等式,或具独立研究价值。这些结果为大规模模型训练中归一化优于截断的实证现象提供了理论解释。
原文摘要 · Abstract (English)
We develop a worst-case complexity theory for stochastically preconditioned stochastic gradient descent (SPSGD) and its accelerated variants under heavy-tailed noise, a setting that encompasses widely used adaptive methods such as Adam, RMSProp, and Shampoo. We assume the stochastic gradient noise has a finite $p$-th moment for some $p \in (1,2]$, and measure convergence after $T$ iterations. While clipping and normalization are parallel tools for stabilizing training of SGD under heavy-tailed noise, there is a fundamental separation in their worst-case properties in stochastically preconditioned settings. We demonstrate that normalization guarantees convergence to a first-order stationary point at rate $\mathcal{O}(T^{-\frac{p-1}{3p-2}})$ when problem parameters are known, and $\mathcal{O}(T^{-\frac{p-1}{2p}})$ when problem parameters are unknown, matching the optimal rates for normalized SGD, respectively. In contrast, we prove that clipping may fail to converge in the worst case due to the statistical dependence between the stochastic preconditioner and the gradient estimates. To enable the analysis, we develop a novel vector-valued Burkholder-type inequality that may be of independent interest. These results provide a theoretical explanation for the empirical preference for normalization over clipping in large-scale model training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。