解释了模型延迟泛化背后的几何机制。
The Geometry of Grokking: Norm Minimization on the Zero-Loss Manifold
- 用约束优化视角，发现训练后期在零损失流形上最小化权重范数。
- 推导出两层网络第一层的解析动态表达式，与实验一致。
- 适合研究深度学习动力学与泛化机制的学者阅读。
Grokking 是神经网络中一种令人困惑的现象：在完全记忆训练数据后，经过长时间延迟才实现完整泛化。先前研究将这种延迟泛化归因于权重衰减驱动的表示学习，但其具体动态仍不清晰。本文提出，后记忆阶段的学习可从约束优化角度理解：梯度下降实际上在零损失流形上最小化权重范数。我们在无穷小学习率和权重衰减系数的极限下严格证明了这一点。为进一步分析该机制，我们引入一个近似方法，将部分参数的学习动态与其他参数解耦。基于此框架，我们推导出两层网络第一层在后记忆阶段的闭式动态表达式。实验表明，使用我们预测的梯度模拟训练过程，能复现 grokking 的延迟泛化与表示学习特征。
原文摘要 · Abstract (English)
Grokking is a puzzling phenomenon in neural networks where full generalization occurs only after a substantial delay following the complete memorization of the training data. Previous research has linked this delayed generalization to representation learning driven by weight decay, but the precise underlying dynamics remain elusive. In this paper, we argue that post-memorization learning can be understood through the lens of constrained optimization: gradient descent effectively minimizes the weight norm on the zero-loss manifold. We formally prove this in the limit of infinitesimally small learning rates and weight decay coefficients. To further dissect this regime, we introduce an approximation that decouples the learning dynamics of a subset of parameters from the rest of the network. Applying this framework, we derive a closed-form expression for the post-memorization dynamics of the first layer in a two-layer network. Experiments confirm that simulating the training process using our predicted gradients reproduces both the delayed generalization and representation learning characteristic of grokking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。