用正则化诱导模型延迟泛化，突破传统欧氏范数限制。
Grokking Beyond the Euclidean Norm of Model Parameters
- 通过稀疏或低秩等性质的正则化，实现延迟泛化
- 深度模型可无显式正则化实现泛化，浅层则不行
- 数据选择即可放大泛化效果，无需调超参
Grokking 指在基于梯度优化的人工神经网络中，过拟合后出现的延迟泛化现象。本文表明，无论是显式还是隐式正则化，均可引发 grokking。具体而言，当存在一种具有性质 P（如稀疏或低秩权重）的模型能在目标任务上泛化时，采用小但非零的性质 P 正则化（如 ℓ₁ 或核范数）的梯度下降，会导致 grokking。这扩展了先前发现：微小的权重衰减即可诱导 grokking。此外，分析显示，增加网络深度使模型可在无显式正则化下实现 grokking 或反 grokking，而浅层结构则无法做到。我们还发现，当模型被正则化至不同于 ℓ₂ 的性质 P 时，ℓ₂ 范数不再是可靠的泛化指标——即便 ℓ₂ 范数增长，模型仍可能泛化。最后，我们证明仅通过数据选择即可放大 grokking 效果，其他超参数保持不变。
原文摘要 · Abstract (English)
Grokking refers to a delayed generalization following overfitting when optimizing artificial neural networks with gradient-based methods. In this work, we demonstrate that grokking can be induced by regularization, either explicit or implicit. More precisely, we show that when there exists a model with a property $P$ (e.g., sparse or low-rank weights) that generalizes on the problem of interest, gradient descent with a small but non-zero regularization of $P$ (e.g., $\ell_1$ or nuclear norm regularization) results in grokking. This extends previous work showing that small non-zero weight decay induces grokking. Moreover, our analysis shows that over-parameterization by adding depth makes it possible to grok or ungrok without explicitly using regularization, which is impossible in shallow cases. We further show that the $\ell_2$ norm is not a reliable proxy for generalization when the model is regularized toward a different property $P$, as the $\ell_2$ norm grows in many cases where no weight decay is used, but the model generalizes anyway. We also show that grokking can be amplified solely through data selection, with any other hyperparameter fixed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。