权重范数决定神经网络泛化延迟时间,呈指数关系。
The Weight Norm Sets the Grokking Timescale: A Causal Delay Law

- 通过固定权重范数,发现泛化延迟随范数增大呈指数增长。
- 范数达到临界值Wc后,延迟时间可被控制在19倍范围内。
- 该规律适用于不同模数,且不受学习率显著影响,适合模型训练调控研究。
Grokking 是指神经网络在完全拟合训练数据后,经过长时间才出现泛化的现象。现有研究对权重范数是否导致这种延迟存在争议:部分研究认为存在一个临界范数,另一些则观察到无固定范数也发生 grokking。本文通过干预训练过程中的权重范数,而非仅观测其变化,解决了这一问题。在带权重衰减的自由训练中,当权重范数达到一个几乎不随种子和学习率变化的临界值 Wc(CV 1-2%)时,网络开始 grok,且 Wc 随模块基数按幂律增长。若将范数钳制在固定的 ρ 倍于 Wc 并保持不变,网络仍会发生 grokking,但延迟时间 T_grok 与 ρ 呈指数关系,即 T_grok ∝ exp(αρ),其中 α ≈ 7.5,适用于四个不同模数(R² = 0.996)。在扫描范围内,保持范数可使延迟时间变化约19倍,而学习率仅影响约2倍;将范数维持在 Wc 以上会延缓而非阻止 grokking。若引入最后一层归一化(LayerNorm),则去除权重尺度与网络函数间的耦合,该指数规律消失;否则指数定律重现。此被约束的范数延迟是自由收缩范数所预测对数延迟的指数对应。
原文摘要 · Abstract (English)
Grokking is the delayed onset of generalization in neural networks, arising long after they fit the training data. Whether the weight norm causes this delay is disputed: some studies report a critical norm at the transition, others observe grokking with no fixed norm at all. We settle this by intervening on the norm during training rather than only observing it. Under free training with weight decay, networks grok when the weight norm reaches a value Wc that varies little across seeds and learning rates (CV 1 to 2 percent) and grows with the modular base as a power law. When we instead clamp the norm to a fixed multiple rho of Wc and hold it there, the network still groks, but the delay follows T_grok proportional to exp(alpha rho). One exponent, alpha near 7.5, fits this delay across four moduli (R^2 = 0.996). Over the swept ranges the held norm moves the delay by about 19x and the learning rate by only about 2x, and holding the norm above Wc slows grokking rather than preventing it. A final LayerNorm removes the dependence by decoupling weight scale from the network function; without it the exponential law returns. This pinned-norm delay is the exponential counterpart to the logarithmic delay predicted for a freely contracting norm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。