揭示梯度下降中神经正切核特征向量的演化规律
Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stability
- 分析边缘稳定状态下神经正切核特征向量的动态变化
- 学习率越大，最终NTK主特征向量越贴近训练目标
- 理论证明两层线性网络中的对齐机制，适用于深度学习优化研究
近年来，深度学习中的神经正切核（NTK）研究日益受到关注。传统上，NTK在训练过程中会主动变化，与特征学习相关。与此同时，近期关于梯度下降（GD）的研究发现一种称为边缘稳定（EoS）的现象：NTK的最大特征值在步长倒数附近振荡。尽管后续工作深入探讨了该特征值行为的内在机制，但对NTK特征向量在EoS下的行为仍缺乏理解。本文详细考察了不同架构下NTK特征向量在EoS中的动态特性。结果表明，较大的学习率会使最终NTK及其完整矩阵的主特征向量与训练目标具有更强的对齐性。进一步通过两层线性网络的理论分析揭示了该现象的内在机制。本研究深化了对深度学习中梯度下降训练动态的理解。
原文摘要 · Abstract (English)
The study of Neural Tangent Kernels (NTKs) in deep learning has drawn increasing attention in recent years. NTKs typically actively change during training and are related to feature learning. In parallel, recent work on Gradient Descent (GD) has found a phenomenon called Edge of Stability (EoS), in which the largest eigenvalue of the NTK oscillates around a value inversely proportional to the step size. However, although follow-up works have explored the underlying mechanism of such eigenvalue behavior in depth, the understanding of the behavior of the NTK eigenvectors during EoS is still missing. This paper examines the dynamics of NTK eigenvectors during EoS in detail. Across different architectures, we observe that larger learning rates cause the leading eigenvectors of the final NTK, as well as the full NTK matrix, to have greater alignment with the training target. We then study the underlying mechanism of this phenomenon and provide a theoretical analysis for a two-layer linear network. Our study enhances the understanding of GD training dynamics in deep learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。