arXiv:2603.13331cs.AIcs.LG2026-03被引 7

揭示了模型延迟泛化的数学规律,可预测何时突然学会通用规律。

The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalization

  • 从权重衰减和学习率出发,建立延迟泛化时间的理论公式。
  • 实验验证:泛化延迟与权重衰减、学习率成反比,与权重比值对数相关。
  • 发现SGD无法实现泛化,而AdamW可做到,关键在解耦记忆与收缩机制。

Grokking——模型在完全记忆训练数据后很久才突然实现泛化的现象——广泛存在但缺乏定量理论解释其延迟时长。本文表明,grokking是正则化训练动力学中的范数驱动表征相变,并提出范数分离延迟定律:$T_{\mathrm{grok}} - T_{\mathrm{mem}} = Θ(γ_{\mathrm{eff}}^{-1} \log(\|θ_{\mathrm{mem}}\|^2 / \|θ_{\mathrm{post}}\|^2))$,其中$γ_{\mathrm{eff}}$为优化器的有效收缩率(SGD中$γ_{\mathrm{eff}} = ηλ$,AdamW中$γ_{\mathrm{eff}} \ge ηλ$)。上界由离散Lyapunov收缩论证得出,下界来自正则化一阶优化的动力学约束。在293次训练中,涵盖模加法、模乘法和稀疏奇偶性任务,验证了三项可证伪预测:权重衰减的反比关系($R^2 = 0.97$)、学习率的反比关系($R^2 = 0.92$)、范数比值的对数依赖(皮尔逊相关$r = 0.91$)。第四项发现:grokking要求优化器能解耦记忆与收缩;在相同超参数下,SGD完全失败,而AdamW可靠实现grokking。结果将grokking重新定义为竞争插值表示间范数分离的可预测后果。进一步推导出一个三输入预测算法,在记忆时刻预测延迟,平均绝对误差34.6%(置信区间[30.0%, 39.4%],$N=60$种子),支持合理提前停止。

原文摘要 · Abstract (English)

Grokking -- the sudden generalisation that appears long after a model has perfectly memorised its training data -- has been widely observed but lacks a quantitative theory explaining the length of the delay. We show that grokking is a norm-driven representational phase transition in regularised training dynamics, and establish the Norm-Separation Delay Law: $T_{\mathrm{grok}} - T_{\mathrm{mem}} = Θ(γ_{\mathrm{eff}}^{-1} \log(\|θ_{\mathrm{mem}}\|^2 / \|θ_{\mathrm{post}}\|^2))$, where $γ_{\mathrm{eff}}$ is the optimiser's effective contraction rate ($γ_{\mathrm{eff}} = ηλ$ for SGD, $γ_{\mathrm{eff}} \ge ηλ$ for AdamW). The upper bound follows from a discrete Lyapunov contraction argument; the matching lower bound from dynamical constraints of regularised first-order optimisation. Across 293 training runs spanning modular addition, modular multiplication, and sparse parity, we confirm three falsifiable predictions: inverse scaling with weight decay ($R^2 = 0.97$), inverse scaling with learning rate ($R^2 = 0.92$), and logarithmic dependence on the norm ratio (Pearson $r = 0.91$). A fourth finding reveals that grokking requires an optimiser capable of decoupling memorisation from contraction: SGD fails entirely at the same hyperparameters where AdamW reliably groks. These results reframe grokking not as a mysterious optimisation artefact but as a predictable consequence of norm separation between competing interpolating representations. We further derive a practical three-input algorithm that predicts grokking delay at memorisation time with 34.6% mean absolute error (bootstrap 95% CI [30.0%, 39.4%], $N=60$ seeds), enabling principled early stopping.

深度学习泛化理论优化机制模型行为

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。