揭示浅层神经网络在漏失激活下临界点分岔的非线性动力学机制
Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLU
- 基于等变梯度度，构建可分析任意神经元数的分岔检测框架
- 发现临界点分岔在α=0时必然出现，且与网络宽度无关
- 适用于研究网络对称性破缺，适合优化理论研究者
本文研究在均方损失和漏失ReLU激活函数下，浅层神经网络的非线性动力学特性。在高斯输入且各层宽度为k的条件下，（1）基于等变梯度度，建立适用于任意神经元数k≥4的理论框架，用于检测随漏失参数α变化时，从全局最小值产生的具有对称性的临界点分岔。分析表明，多模退化现象在临界点α=0处始终出现，与k无关。（2）进一步证明该分岔与网络宽度无关，仅出现在α≥0时，且在工程范围α∈(0,1)内，全局最小值不会发生进一步对称性破缺不稳定性。通过k=5的具体例子展示了该框架及对应的分岔与对称性结构。
原文摘要 · Abstract (English)
In this work, we study the nonlinear dynamics of a shallow neural network trained with mean-squared loss and leaky ReLU activation. Under Gaussian inputs and equal layer width k, (1) we establish, based on the equivariant gradient degree, a theoretical framework, applicable to any number of neurons k>= 4, to detect bifurcation of critical points with associated symmetries from global minimum as leaky parameter $α$ varies. Typically, our analysis reveals that a multi-mode degeneracy consistently occurs at the critical number 0, independent of k. (2) As a by-product, we further show that such bifurcations are width-independent, arise only for nonnegative $α$ and that the global minimum undergoes no further symmetry-breaking instability throughout the engineering regime $α$ in range (0,1). An explicit example with k=5 is presented to illustrate the framework and exhibit the resulting bifurcation together with their symmetries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。