通过训练雅可比矩阵揭示神经网络优化的几何结构。
Understanding Gradient Descent through the Training Jacobian
- 用初始值对参数的雅可比矩阵分析训练过程几何。
- 雅可比奇异值分三区：混沌、近1的主干、稳定区，主干方向扰动几乎不变。
- 主干子空间在不同随机种子间重叠大，适合研究泛化与鲁棒性。
我们通过训练后网络参数对初始值的雅可比矩阵，研究神经网络训练的几何特性。分析发现训练过程具有低维结构，依赖输入数据但基本不受标签影响。雅可比矩阵的奇异值谱包含三个显著区域：数值远大于1的‘混沌’区、接近1的‘主干’区、小于1的‘稳定’区。在每个主干方向上，左右奇异向量几乎相同，表明初始扰动在训练中几乎不变地传递。这些扰动对分布内输出几乎无影响，但在分布外有显著作用。尽管雅可比仅局部有效，但不同随机种子间的主干子空间存在显著重叠。代码已开源：https://github.com/EleutherAI/training-jacobian。
原文摘要 · Abstract (English)
We examine the geometry of neural network training using the Jacobian of trained network parameters with respect to their initial values. Our analysis reveals low-dimensional structure in the training process which is dependent on the input data but largely independent of the labels. We find that the singular value spectrum of the Jacobian matrix consists of three distinctive regions: a "chaotic" region of values orders of magnitude greater than one, a large "bulk" region of values extremely close to one, and a "stable" region of values less than one. Along each bulk direction, the left and right singular vectors are nearly identical, indicating that perturbations to the initialization are carried through training almost unchanged. These perturbations have virtually no effect on the network's output in-distribution, yet do have an effect far out-of-distribution. While the Jacobian applies only locally around a single initialization, we find substantial overlap in bulk subspaces for different random seeds. Our code is available at https://github.com/EleutherAI/training-jacobian
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。