解释神经网络损失曲面中大量接近零的海森矩阵特征值来源。
Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks
- 从参数化对称性破缺角度，揭示近零特征值由弱扰动的伪戈尔德斯通模式构成。
- 在深度线性网络中精确构造零特征值对应的特征向量，引入ReLU后形成近零模式。
- 该机制适用于全连接与卷积结构，在真实模型（如CIFAR-10）中得到验证。
海森矩阵刻画了训练损失的局部几何结构，尽管其最大特征值已有解释，但大量趋近于零的特征值成因仍不明确。本文提出，这些特征值主要来自网络参数化中连续对称性的弱破缺所引发的伪戈尔德斯通模式。在深度线性网络中，这些对称性是精确的，产生平坦方向和精确零特征值，其特征向量可显式构造。引入ReLU非线性作为微小扰动后，对称性被弱破坏，导致特征值接近零。通过分析特征向量谱，发现高曲率方向正交于对称子空间，而大部分特征值几乎完全位于该子空间内。该机制在两层ReLU学生-教师模型和一个在CIFAR-10上训练的网络中得到验证，卷积结构示例表明该诊断方法可推广至非全连接层。结果将海森矩阵的主体特征值与弱对称性破缺联系起来，澄清了近零模式的起源。
原文摘要 · Abstract (English)
The Hessian of the training loss governs the local geometry of the loss landscape, yet despite existing explanations for its largest eigenvalues, the origin of the vast multitude of vanishingly small eigenvalues remains elusive. We argue that the bulk consists of the weakly lifted pseudo-Goldstone modes of the continuous symmetries of the network parametrization. In deep linear networks these symmetries are exact: they generate flat directions and hence exact zero modes, whose eigenvectors we construct explicitly. Introducing a ReLU nonlinearity as a perturbation, we show that it breaks these symmetries weakly and explicitly. Resolving the spectrum at the level of eigenvectors, we find that the high-curvature directions are orthogonal to the symmetry subspace, while the bulk lies almost entirely within it. We demonstrate the mechanism in a two-layer ReLU student--teacher model and in a network trained on CIFAR-10. A convolutional example demonstrates that the same diagnostic extends beyond fully connected layers. Together, these results link the Hessian bulk to weakly broken symmetries and clarify the origin of near-zero modes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。