动量让随机梯度下降在不同批量下呈现两种稳定态，影响模型平坦性。
Momentum Further Constrains Sharpness at the Edge of Stochastic Stability
- 动量使优化在小批量时放大随机波动，倾向更平坦解；大批量时恢复经典稳定作用，倾向更尖锐解。
- 小批量下梯度曲率收敛至 $2(1-β)/η$，大批量下收敛至 $2(1+β)/η$，与动量系数 β 相关。
- 揭示动量与批量的协同效应，对超参调优有指导意义，适合研究优化机制的学者。
近期研究表明，（随机）梯度下降会自我组织于不稳定性边界附近，影响优化过程与解的性质。动量和小批量梯度广泛用于实际深度学习优化，但其是否处于类似的不稳定性范围仍不明确。本文证明，带动量的 SGD 展现出类边缘随机稳定性（EoSS）行为，其表现依赖于批量大小，无法由单一动量调整的稳定性阈值解释。批量曲率（即期望方向上的小批量曲率）在两个不同区间趋于稳定：小批量时收敛至较低平台 $2(1-β)/η$，反映动量对随机波动的放大作用，倾向于更平坦区域；大批量时收敛至更高平台 $2(1+β)/η$，动量恢复经典稳定效应，倾向更尖锐区域，与全批量动态一致。该结果与线性稳定性阈值吻合，并讨论了对超参数调优与耦合关系的启示。
原文摘要 · Abstract (English)
Recent work suggests that (stochastic) gradient descent self-organizes near an instability boundary, shaping both optimization and the solutions found. Momentum and mini-batch gradients are widely used in practical deep learning optimization, but it remains unclear whether they operate in a comparable regime of instability. We demonstrate that SGD with momentum exhibits an Edge of Stochastic Stability (EoSS)-like regime with batch-size-dependent behavior that cannot be explained by a single momentum-adjusted stability threshold. Batch Sharpness (the expected directional mini-batch curvature) stabilizes in two distinct regimes: at small batch sizes it converges to a lower plateau $2(1-β)/η$, reflecting amplification of stochastic fluctuations by momentum and favoring flatter regions than vanilla SGD; at large batch sizes it converges to a higher plateau $2(1+β)/η$, where momentum recovers its classical stabilizing effect and favors sharper regions consistent with full-batch dynamics. We further show that this aligns with linear stability thresholds and discuss the implications for hyperparameter tuning and coupling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。