提出非欧梯度下降的稳定性边界理论，统一解释多种优化方法的震荡现象。
Non-Euclidean Gradient Descent Operates at the Edge of Stability
- 用方向光滑性解释非欧梯度下降的稳定边界行为
- 在多种非欧优化中观察到尖度逼近并围绕2/η震荡
- 为多类优化器提供可通用的谱诊断工具
边缘稳定性（Edge of Stability, EoS）指梯度下降过程中海森矩阵最大特征值趋近并维持在稳定性阈值 $2/η$ 附近的现象。尽管违背经典光滑性假设，该现象在深度学习中广泛存在，但理论基础仍不完善。本文通过方向光滑性视角阐释EoS，并将其推广至任意范数下的非欧梯度下降，定义广义尖度度量。该度量涵盖经典梯度下降、预条件梯度下降，以及此前未研究过的方法如 $\ell_{\infty}$-descent、块坐标下降（Block CD）、谱梯度下降（Spectral GD）及其归一化版本。在神经网络实验中，非欧梯度下降同样表现出尖度渐进增大后围绕或超过阈值 $2/η$ 的振荡行为。本框架提供了适用于多种非欧优化方法的几何感知谱诊断工具。
原文摘要 · Abstract (English)
The Edge of Stability (EoS) is a phenomenon where the sharpness (largest eigenvalue) of the Hessian approaches and then hovers near the stability threshold $2/η$ during gradient descent (GD) with step size $η$. Despite (apparently) violating classical smoothness assumptions, EoS has been widely observed in deep learning, but its theoretical foundations remain incomplete. We provide an interpretation of EoS through the lens of Directional Smoothness [Mishkin et al., 2024]. This interpretation naturally extends to non-Euclidean norms, which we use to define generalized sharpness under an arbitrary norm. Our generalized sharpness measure includes previously studied vanilla GD and preconditioned GD as special cases, as well as methods for which EoS has not been studied, such as $\ell_{\infty}$-descent, Block CD, Spectral GD, and their normalized versions. Through experiments on neural networks, we show that non-Euclidean GD with our generalized sharpness also exhibits progressive sharpening followed by oscillations around or above the threshold $2/η$. Practically, our framework provides a geometry-aware spectral diagnostic that can be applied across a broad class of non-Euclidean gradient methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。