用分岔理论解释梯度下降在稳定性边缘的运行机制。
A Bifurcation Theory Framework for Gradient Descent on the Edge of Stability
- 将训练分解为法向与切向动态，分析稳定性来源。
- 证明在边缘稳定状态下可收敛至最小值流形。
- 统一了已有成果，适合研究深度学习优化的学者。
边缘稳定性（Edge of Stability, EoS）现象指梯度下降在尖锐度超过经典收敛阈值的情况下仍能长期降低损失，广泛存在于现代深度学习中，但在真实场景下仍缺乏理解。以往严格分析多限于标量或低维损失且具有特定结构。本文针对过参数化神经网络，构建了梯度下降在边缘稳定性下的分岔理论框架。通过将训练动态分解为最小值流形的法向与切向分量，我们证明：稳定的EoS训练源于法向方向的翻转分岔，由一阶李雅普诺夫系数符号决定；而切向动态则趋向尖锐度下降区域。在损失景观的谱和几何假设下，我们证明在EoS阈值训练时可收敛至最小值流形。作为推论，我们发现Gan (2026)提出的乘积稳定性条件是本框架的特例。
原文摘要 · Abstract (English)
The Edge of Stability (EoS) phenomenon, where gradient descent operates with sharpness exceeding the classical convergence threshold yet the loss decreases over long timescales, is ubiquitous in modern deep learning but remains poorly understood in realistic settings. Prior rigorous analyses have been largely confined to scalar or low-dimensional losses with specific structural forms. In this work, we develop a bifurcation theory framework for gradient descent on the edge of stability that applies directly to overparameterized neural networks. By decomposing the training dynamics into components normal and tangent to the manifold of minimizers, we show that stable EoS training arises from a flip bifurcation in the normal direction, governed by the sign of the first Lyapunov coefficient, while the tangent dynamics drift toward regions of decreasing sharpness. Under mild spectral and geometric assumptions on the loss landscape, we prove convergence to the minimizing manifold when training at the EoS threshold. As a corollary, we recover and unify prior results: we show that the product-stability condition of Gan (2026) is an instance of our framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。