arXiv:2606.18080cs.LG2026-06被引 4

提出可预测的连续时间模型,解释深度学习梯度下降在稳定边缘的动态行为。

Edge Flow: A Tractable and Predictive Continuous-Time Model for Gradient Descent at the Edge of Stability

  • 通过三个耦合微分方程分解动力学为中心、振荡方向与幅度。
  • 模型能准确追踪梯度下降在稳定边缘的震荡现象,且只需两次梯度计算。
  • 适用于研究和缓解深度学习训练中的不稳定性问题。

深度学习中的梯度下降可能运行在稳定边缘(EoS)区域,此时损失函数的海森矩阵最大特征值接近稳定性阈值 $2/η$($η$ 为学习率)。经典分析工具如梯度流和下降引理在此不适用,因此亟需一种适用于 EoS 的连续时间模型。本文提出 Edge Flow,一个由三个耦合常微分方程组成的系统,可精确建模梯度下降在 EoS 下的动力学。该模型将动态分解为中心(遵循对称化损失上的修正梯度流)、振荡方向(通过瑞利商动态追踪海森矩阵主特征向量)和振幅(根据尖锐度是否超过 $2/η$ 指数增长或衰减)。关键的是,尖锐度稳定由耦合动力学中自稳定反馈环实现。离散化 Edge Flow 每步仅需两次梯度评估和一次海森-向量乘积。实验表明,Edge Flow 在追踪梯度下降动态上至少等同于已有 EoS 模型,并能解析地捕捉到尖锐度在进入 EoS 初期的振荡行为,同时为理解与缓解该区域不稳定性提供理论框架。

原文摘要 · Abstract (English)

Gradient descent in deep learning may operate at the edge of stability (EoS), a regime in which the largest eigenvalue of the loss Hessian hovers near the stability threshold $2/η$, where $η$ is the learning rate. Classical analysis tools such as gradient flow and the descent lemma do not apply here, motivating the search for a continuous-time model valid at EoS. We propose Edge Flow, a system of three coupled ordinary differential equations that provides a tractable, faithful, and predictive model of gradient descent dynamics at EoS. Edge Flow decomposes the dynamics into a center, an oscillation direction, and an oscillation magnitude. The center follows a modified gradient flow on a symmetrized loss; the direction tracks a top eigenvector of the Hessian via Rayleigh quotient dynamics; and the magnitude grows or decays exponentially depending on whether the sharpness exceeds or falls below the threshold $2/η$. Crucially, sharpness stabilization emerges from the coupled dynamics via a self-stabilization feedback loop. Discretizing Edge Flow only requires two gradient evaluations and one Hessian--vector product at each iteration. We demonstrate empirically that Edge Flow tracks the dynamics of gradient descent at least as faithfully as previously proposed continuous-time EoS models, while in addition resolving the oscillation of the sharpness at the onset of EoS, and that it provides a principled framework for understanding and mitigating instabilities in this regime.

优化理论稳定边缘连续模型梯度下降

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。