arXiv:2511.10362cs.LGcs.SY2025-11中稿 · publication in SIA…综述被引 2

解析深度线性网络梯度流的动态与损失景观,揭示其数学结构与稳定性机制。

Gradient Flow Equations for Deep Linear Neural Networks: A Survey from a Network Perspective

  • 从邻接矩阵视角建模梯度流,揭示其为满足特定代数性质的微分方程组。
  • 损失函数无局部极小值,存在无穷多全局极小与鞍点,且临界值对应数据奇异值学习量。
  • 提出商空间结构,可统一表示相同损失值的临界点,并直接确定鞍点稳定流形。

本文综述了深度线性神经网络(即无激活函数、使用平方损失的深层网络)在梯度下降极限下(步长趋于0)的梯度流方程的动力学与损失景观的最新进展。当以网络邻接矩阵形式表述时,这些梯度流构成一类收敛的矩阵常微分方程,具有幂零性、多项式性、等谱性及守恒律。损失景观被详细刻画:存在无穷多个全局极小值和鞍点(包括严格与非严格),但无局部极小或极大值。损失函数本身是正半定的李雅普诺夫函数,其等高集为临界点的无界不变集,临界值对应梯度轨迹沿特定路径所学习的数据输入-输出奇异值数量。本文采用的邻接矩阵表示揭示了商空间结构——每个损失临界值在商空间中唯一表示,其余同值临界点均属于该结构的纤维;同时,即使海森矩阵失效,也能轻易确定鞍点的稳定与不稳定子流形。

原文摘要 · Abstract (English)

The paper surveys recent progresses in understanding the dynamics and loss landscape of the gradient flow equations associated to deep linear neural networks, i.e., the gradient descent training dynamics (in the limit when the step size goes to 0) of deep neural networks missing the activation functions and subject to quadratic loss functions. When formulated in terms of the adjacency matrix of the neural network, as we do in the paper, these gradient flow equations form a class of converging matrix ODEs which is nilpotent, polynomial, isospectral, and with conservation laws. The loss landscape is described in detail. It is characterized by infinitely many global minima and saddle points, both strict and nonstrict, but lacks local minima and maxima. The loss function itself is a positive semidefinite Lyapunov function for the gradient flow, and its level sets are unbounded invariant sets of critical points, with critical values that correspond to the amount of singular values of the input-output data learnt by the gradient along a certain trajectory. The adjacency matrix representation we use in the paper allows to highlight the existence of a quotient space structure in which each critical value of the loss function is represented only once, while all other critical points with the same critical value belong to the fiber associated to the quotient space. It also allows to easily determine stable and unstable submanifolds at the saddle points, even when the Hessian fails to obtain them.

深度线性网络梯度流损失景观微分方程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。