arXiv:2410.13866q-bio.NCcs.AI2024-10ICLR被引 1

解决神经网络中死单元导致能量函数失效的问题,提出新方法确保稳定性和吸引力域分析。

Associative memory and dead neurons

  • 通过分析能量函数的海森矩阵,从死单元的平坦区域恢复状态信息
  • 证明死单元所在平坦区域的稳定点会整体成为吸引子
  • 设计无平坦区的能量函数,适用于非对称连接等复杂结构

在《神经生物学与机器学习中的大型关联记忆问题》中,Dmitry Krotov 和 John Hopfield 提出了一种系统构造具有非增能量或李雅普诺夫函数的神经微分方程的方法。本文研究该能量函数,发现其易受死神经元问题影响:状态空间中每个神经元死亡的点都位于一个能量恒定的非紧致区域。在此类平坦区域,仅靠能量函数无法完全确定所有自由度,因而不能用于分析稳定性、寻找稳态或吸引子区域。我们对动力系统进行直接分析,提出解决方案:(i) 稳态点的状态向量信息可由能量和拉格朗日函数的海森矩阵完全提取;(ii) 稳定性分析只需限于海森矩阵的范围;(iii) 若触及平坦区域的稳态是稳定的,则整个平坦区域即为吸引子。由于真实架构中海森矩阵分析复杂,我们进一步提出一种稍作修改的动力系统,保持稳态结构不变,但可导出多种无死单元对应平坦区的李雅普诺夫函数。这些能量函数允许使用非正定海森矩阵,甚至支持非对称前馈与反馈连接的架构。

原文摘要 · Abstract (English)

In "Large Associative Memory Problem in Neurobiology and Machine Learning," Dmitry Krotov and John Hopfield introduced a general technique for the systematic construction of neural ordinary differential equations with non-increasing energy or Lyapunov function. We study this energy function and identify that it is vulnerable to the problem of dead neurons. Each point in the state space where the neuron dies is contained in a non-compact region with constant energy. In these flat regions, energy function alone does not completely determine all degrees of freedom and, as a consequence, can not be used to analyze stability or find steady states or basins of attraction. We perform a direct analysis of the dynamical system and show how to resolve problems caused by flat directions corresponding to dead neurons: (i) all information about the state vector at a fixed point can be extracted from the energy and Hessian matrix (of Lagrange function), (ii) it is enough to analyze stability in the range of Hessian matrix, (iii) if steady state touching flat region is stable the whole flat region is the basin of attraction. The analysis of the Hessian matrix can be complicated for realistic architectures, so we show that for a slightly altered dynamical system (with the same structure of steady states), one can derive a diverse family of Lyapunov functions that do not have flat regions corresponding to dead neurons. In addition, these energy functions allow one to use Lagrange functions with Hessian matrices that are not necessarily positive definite and even consider architectures with non-symmetric feedforward and feedback connections.

神经网络能量函数稳定性分析死单元

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。