提出随机稳定性边缘新机制，解释小批量SGD为何更优
Edge of Stochastic Stability: Revisiting the Edge of Stability for SGD
- 用批梯度方向曲率替代传统海森矩阵最大特征值作为稳定指标
- 小批量下最大特征值被抑制，与扁平极小值现象一致
- 为理解小批量训练的泛化优势提供新数学框架
Cohen等人（2021）发现，使用全批量梯度下降且步长为η时，神经网络的全批量海森矩阵最大特征值λ_max始终稳定在2/η附近。这一现象对收敛性和泛化性有重要影响。然而，该规律不适用于小批量优化算法，限制了其普适性。本文揭示，小批量随机梯度下降（SGD）实际上运行于一个我们称为随机稳定性边缘（EoSS）的新范式中。在此范式下，稳定在2/η的是‘批尖锐度’：即小批量海森矩阵沿对应随机梯度方向的期望曲率。由于λ_max通常小于批尖锐度，因此被抑制，这与长期观察到的现象一致——较小批量和较大步长更倾向于找到平坦的极小值。本文进一步讨论了该发现对建模SGD轨迹的数学意义。
原文摘要 · Abstract (English)
Recent findings by Cohen et al., 2021, demonstrate that when training neural networks using full-batch gradient descent with a step size of $η$, the largest eigenvalue $λ_{\max}$ of the full-batch Hessian consistently stabilizes around $2/η$. These results have significant implications for convergence and generalization. This, however, is not the case for mini-batch optimization algorithms, limiting the broader applicabilityof the consequences of these findings. We show mini-batch Stochastic Gradient Descent (SGD) trains in a different regime we term Edge of Stochastic Stability (EoSS). In this regime, what stabilizes at $2/η$ is Batch Sharpness: the expected directional curvature of mini-batch Hessians along their corresponding stochastic gradients. As a consequence $λ_{\max}$ -- which is generally smaller than Batch Sharpness -- is suppressed, aligning with the long-standing empirical observation that smaller batches and larger step sizes favor flatter minima. We further discuss implications for mathematical modeling of SGD trajectories.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。