解释了神经网络训练中为何梯度下降会稳定在临界曲率处。
The Origin of Edge of Stability

- 通过构造边耦合函数,推导出步长递推关系和损失变化公式。
- 理论证明曲率会精确趋近于 2/η,无误差间隙。
- 揭示了固定点与周期-2轨道的形成机制,适用于理解训练动态。
全批量梯度下降在神经网络上使最大Hessian特征值趋近于阈值 2/η(η为学习率)。这一现象——边缘稳定性——长期缺乏统一解释:已有研究仅说明接近边缘时的自调节行为,却未说明为何轨迹从任意初始化出发都会被强制推向 2/η。本文提出“边耦合”函数,其系数由梯度下降更新唯一确定。对临界条件求差得一步递推式,其稳定边界恰为 2/η;二阶展开后得到损失变化公式,其望远镜和迫使曲率逼近 2/η。两个公式涉及不同Hessian平均,但均通过中值定理局部化至步长段内某点的真实Hessian,实现无间隙的精确驱动。令边耦合的两个梯度为零,可分类固定点与周期-2轨道;在固定点附近,问题退化为仅依赖半振幅的函数,从而确定支持周期-2轨道的方向及其在临界学习率的哪一侧出现。
原文摘要 · Abstract (English)
Full-batch gradient descent on neural networks drives the largest Hessian eigenvalue to the threshold $2/η$, where $η$ is the learning rate. This phenomenon, the Edge of Stability, has resisted a unified explanation: existing accounts establish self-regulation near the edge but do not explain why the trajectory is forced toward $2/η$ from arbitrary initialization. We introduce the edge coupling, a functional on consecutive iterate pairs whose coefficient is uniquely fixed by the gradient-descent update. Differencing its criticality condition yields a step recurrence with stability boundary $2/η$, and a second-order expansion yields a loss-change formula whose telescoping sum forces curvature toward $2/η$. The two formulas involve different Hessian averages, but the mean value theorem localizes each to the true Hessian at an interior point of the step segment, yielding exact forcing of the Hessian eigenvalue with no gap. Setting both gradients of the edge coupling to zero classifies fixed points and period-two orbits; near a fixed point, the problem reduces to a function of the half-amplitude alone, which determines which directions support period-two orbits and on which side of the critical learning rate they appear.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。