研究大步长梯度下降在平坦极小值流形附近的动态行为,揭示其收敛机制。
Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima

- 提出大步长梯度下降在平坦极小值流形邻域的统一归一化形式。
- 证明在向量输出和流形极小值情况下仍可保证收敛。
- 适用于深度矩阵分解,揭示极小值集为球面乘积上的纤维丛结构。
梯度下降(GD)理论中一个重要量是尖锐性(sharpness),定义为目标函数黑塞矩阵的最大特征值。经典分析通常要求步长严格小于尖锐性的两倍倒数,但这一条件在深度神经网络训练中常被违反。近期工作在过参数化最小二乘问题且单标量输出情形下,给出了大步长GD在孤立平坦极小值邻域的归一化形式,并建立了三个收敛结果。本文将该理论扩展至两类更一般情形:(1) 过参数化最小二乘问题中的向量输出(包括任意观测数量的回归);(2) 平坦极小值流形邻域(我们证明这对矩阵分解等应用至关重要)。本文推广了归一化形式及全部三项收敛定理,克服了多个技术难点,包括通过新方法求解奇异偏微分方程,该方法或具独立价值。进一步证明,该框架在温和假设下适用于深度矩阵分解,得出若干新结构性结果:平坦极小值集构成球面乘积上的纤维丛,且尖锐性沿该流形为Morse-Bott型。
原文摘要 · Abstract (English)
An important quantity in the theory of gradient descent (GD) is the \emph{sharpness}, defined as the largest eigenvalue of the objective Hessian. Classical analyses typically require the step size to be uniformly smaller than twice the reciprocal of the sharpness, but this condition is frequently violated in the training of deep neural networks. Recent work bridges this gap in the setting of overparametrised least-squares with a \emph{single scalar output}, providing a normal form for large-step GD in a neighbourhood of an \emph{isolated} flat minimum and establishing three corresponding convergence results. In this paper, we extend this theory in two directions: (1) to overparametrised least-squares with \emph{vector-valued outputs} (including regression with arbitrarily many observations), and (2) to a neighbourhood of a \emph{manifold} of flat minima (which we show is essential for applications such as matrix factorisation). We generalise both the normal form and all three convergence theorems of \cite{macdonaldeos} to this broader setting, overcoming several technical challenges, including the solution of a singular partial differential equation via a novel method that may be of independent interest. We further show that our framework applies to deep matrix factorisation under mild assumptions, yielding several new structural results. In particular, we prove that the set of flat minima forms a fibre bundle over a product of spheres, and that the sharpness is Morse-Bott along this manifold.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。