首次定量预测AdamW下模型的grokking延迟时间。
First-Passage Prediction of Grokking Delay: ACalibrated Law under AdamW with Causal Validation
- 将延迟视为首次通过时间，推导出闭式预测公式。
- 单组超参数校准可跨41倍延迟范围预测26次实验，平均误差17.7%。
- 适用于小规模算法任务，对架构敏感但具可解释性，适合研究优化机制者。
我们首次在AdamW下对grokking延迟给出了定量预测。将延迟视为首次通过时间，推导出闭式公式：T_grok - T_mem = (1 / 2 kappa_LL eta lambda) log(V_mem / V_star)，其中V_t = ||theta_t||^2为参数平方范数，V_star为依赖架构的阈值，kappa_LL吸收了AdamW对纯净SGD收缩率2 eta lambda的修正。在单一超参数单元上校准(kappa_LL, V_star)，可对26个保留运行预测grokking延迟，平均绝对百分比误差（MAPE）为17.7%，覆盖41倍延迟范围；该规律推广至MLPs（MAPE 18.0%，N=34），在跨任务扩展中退化至23.3%（N=46，43.5倍范围），残差结构显示同一架构内V_star / V_mem相对稳定（1L Transformer CV约14%）。首次通过V_t是必要非充分条件。量化边界定理表明正延迟需同时满足范数分离（V_mem > V_post）与角度可达性（alpha_star = arcsin(C / V_T_mem^(1/2))），其中C由经验NTK特征映射和验证边际分位数计算得出。在模p=89上校准C，预测p=97时alpha_star=47.2°（实测47.8°，误差1.3%），实现跨单元先验预测。因果干预冻结范数或移除记忆阶段权重衰减会消除grokking（0/6 vs. 3/3基线），使角度位移锁定在约12度。kappa_LL为每架构经验测量，而非由(β₁, β₂, ε)推导，同架构内部变异系数不超过15%，但不同架构间差异约2倍（超越深度因素）。实证范围为算法任务（模运算、稀疏奇偶性）下的AdamW；该规律是否适用于自然语言规模模型尚待验证。
原文摘要 · Abstract (English)
We give the first quantitative prediction of grokking delay under AdamW. Treating the delay as a first-passage time, we derive a closed-form law T_grok - T_mem = (1 / 2 kappa_LL eta lambda) log(V_mem / V_star), where V_t = ||theta_t||^2 is the squared parameter norm, V_star is an architecture-dependent threshold, and kappa_LL absorbs the AdamW correction to the clean-SGD contraction rate 2 eta lambda. Calibrating (kappa_LL, V_star) on a single hyperparameter cell predicts grokking delays on 26 held-out runs with MAPE 17.7% over a 41x delay range; the law generalises to MLPs (MAPE 18.0%, N=34) and degrades to 23.3% on cross-task extension (N=46, 43.5x range), with a structured residual in which V_star / V_mem stays comparatively stable within architecture (CV about 14% on the 1L transformer). First-passage of V_t is necessary but not sufficient. A quantile-margin theorem establishes that positive delay requires both norm separation V_mem > V_post and angular reachability of a threshold alpha_star = arcsin(C / V_T_mem^(1/2)), where C is computable from the empirical NTK feature map and the validation-margin quantile. Calibrating C on modulus p=89 predicts alpha_star = 47.2 degrees at p=97 (observed 47.8 degrees, error 1.3%) as a prior cross-cell prediction. Causal interventions that freeze the norm or remove weight decay at memorisation eliminate grokking (0/6 vs. 3/3 baseline), trapping the angular displacement near 12 degrees. kappa_LL is empirically measured per architecture rather than derived from (beta_1, beta_2, epsilon); within-architecture CV stays at most 15% across four architectures, but values differ by about 2x between architectural variants beyond depth alone. Empirical scope is algorithmic tasks (modular arithmetic, sparse parity) under AdamW; whether the law transfers to natural-language scale models is open.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。