通过分层海森矩阵分析神经网络局部几何，揭示过拟合与泛化的关系。
Local properties of neural networks through the lens of layer-wise Hessians
- 用每层参数的二阶导矩阵刻画网络局部几何结构
- 发现训练中分层海森谱分布具有稳定规律，与泛化性能相关
- 适合研究模型优化、架构设计和训练稳定性的人参考
我们提出一种通过分层海森矩阵分析神经网络的方法。每个功能模块（层）的局部海森矩阵定义为标量函数对本层参数的二阶导数矩阵，提供表征参数空间局部几何的正式工具。我们发现局部海森矩阵的谱特性（如特征值分布）可量化反映过拟合、欠参数化与模型表达力。通过在37个数据集上开展111组实验的广泛实证研究，结果表明局部海森矩阵在训练过程中的演化呈现一致的结构性规律，并且其谱与泛化性能存在显著相关性。这些发现为利用局部几何分析指导深度神经网络的诊断与设计奠定了基础，将优化几何与功能行为联系起来，为改进网络架构与训练稳定性提供了实用洞见。
原文摘要 · Abstract (English)
We introduce a methodology for analyzing neural networks through the lens of layer-wise Hessian matrices. The local Hessian of each functional block (layer) is defined as the matrix of second derivatives of a scalar function with respect to the parameters of that layer. This concept provides a formal tool for characterizing the local geometry of the parameter space. We show that the spectral properties of local Hessians, such as the distribution of eigenvalues, reveal quantitative patterns associated with overfitting, underparameterization, and expressivity in neural network architectures. We conduct an extensive empirical study involving 111 experiments across 37 datasets. The results demonstrate consistent structural regularities in the evolution of local Hessians during training and highlight correlations between their spectra and generalization performance. These findings establish a foundation for using local geometric analysis to guide the diagnosis and design of deep neural networks. The proposed framework connects optimization geometry with functional behavior and offers practical insight for improving network architectures and training stability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。