研究常步长SGD在平坦最小值处的长期行为，发现其收敛尺度和分布随平坦度变化。
Scaling Limits of Constant-Stepsize SGD at Flat Minima
- 针对具有平坦最小值的凸目标函数，分析常步长SGD的不变分布特性。
- 当最小值平坦度指数m>2时，迭代点集中在α^{1/m}尺度，分布非高斯。
- 适用于理解深度学习优化中平坦最小值对泛化的影响，适合优化理论研究者。
对于常步长α的随机梯度下降（SGD），在长时间尺度下，以极小值为中心的迭代点的不变分布描述了算法行为。在强凸情况下，该不变分布具有√α的缩放性质，并在α↓0时趋于高斯分布。本文研究在具有平坦最小值和（次）二次尾部的凸目标函数H上，由收缩驱动链生成的马尔可夫噪声下的SGD。对任意足够小的常步长α，证明了在依赖于α的Wasserstein距离下，存在唯一且几何收敛的增强型不变分布。当极小值点x⋆的局部平坦度指数m≥2（即∇²H(x) ≍ ∥x−x⋆∥^{m−2}I_d，当x→x⋆）时，得到收缩因子为1−cα^{m−1}的界，其中c>0为常数，这在m=2时退化为1−cα。随后分析小步长极限：不变分布集中在α^{1/m}尺度，重缩放后的迭代点弱收敛于随机微分方程 dY_t = −h₀(Y_t)dt + Σ^{1/2}dB_t 的平稳分布，其中h₀为极小值处的极限漂移，Σ为渐近协方差。当m=2时恢复高斯极限，m>2时给出一般非高斯平稳极限。最后，给出了坐标可分离目标函数且各维度平坦度指数不同时的对应结果。
原文摘要 · Abstract (English)
For stochastic gradient descent (SGD) with a constant stepsize $α$, the invariant law of the iterates, centered at a minimizer, describes the behavior of the algorithm over long time horizons. In the strongly convex case, this invariant law has the familiar $\sqrtα$ scaling and a Gaussian limit as $α\downarrow 0$. We show that this behavior changes fundamentally for convex objectives $H$ with flat minima and (sub)quadratic tails. More specifically, we study SGD with Markovian noise generated by a contractive driving chain. For every sufficiently small constant stepsize $α$, we prove existence, uniqueness, and geometric convergence to an augmented invariant law in a Wasserstein distance induced by an $α$-dependent metric. When the minimizer $x_\star$ has local flatness exponent $m\ge2$, meaning that $\nabla^2 H(x)\asymp \lVert x-x_\star\rVert^{m-2} I_d$ as $x\to x_\star$, we obtain a contraction bound with factor $1-cα^{m-1}$, where $c>0$ is a constant. This recovers the factor $1-cα$ in the quadratic case $m=2$. We then analyze the small-stepsize scaling limit. We show that the invariant law concentrates on the scale $α^{1/m}$ and that the rescaled iterates converge weakly to the stationary distribution of the stochastic differential equation $$ dY_t=-h_0(Y_t)\,dt+Σ^{1/2}\,dB_t , $$ where $h_0$ is the limiting drift at the minimizer and $Σ$ denotes the asymptotic covariance. This recovers the Gaussian limit when $m=2$ and gives generally non-Gaussian stationary limits in the flat case $m>2$. Finally, we give corresponding results for coordinate-separable objectives with unequal flatness exponents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。