通过结构化先验干预，证明泛化延迟本质是形成正确表征所需时间。
Structure-Specific Representational Priors Causally Control the Grokking Delay
- 用对比损失注入不同结构先验，控制变量验证表征结构影响
- 仅真实结构能加速泛化，最快提升22倍，且与权重范数相关
- 通过范数钳制可稳定加速，适合需快速泛化的模型训练场景
Grokking——在训练集过拟合后仍长期延迟实现泛化——已被无结构干预（如梯度过滤、权重范数钳制、几何惩罚）加速。但这一延迟是否特指形成任务结构化表征所需时间，此前仅为观察。我们通过因果实验，在单层Transformer学习模加法任务时，利用监督对比损失注入三种内容不同的表征先验：(i) 真实结构（(a+b) mod p），(ii) 逻辑一致但错误的兄弟结构（(a−b) mod p），(iii) 随机划分（仅可记忆）。三者具有相同损失形式、强度、类别规模和几何结构。结果显示泛化出现概率呈清晰梯度：真实结构22/30次成功，兄弟结构14/15次，随机结构0/20次（Fisher检验p=1.3×10⁻⁷）。一个与权重范数匹配的对照实验在交叉熵上重现范数轨迹，泛化率为0/15，排除范数为中介。探测显示结构形成早于且可预测泛化。只有真实结构能加速grokking（最高达2.75倍），且具有剂量依赖性和双峰特征。进一步预测验证：因加速受权重范数副作用调控，训练中钳制范数可实现稳定、独立的加速，中位速度提升8.6倍（最快种子达22倍，1000轮内），并随范数降低单调增长；残差停滞现象也消失（整体比较显著，0/40对6/20，p=7.7×10⁻⁴），但非单一方法显著。因此，grokking延迟因果上即为形成正确表征结构的时间，由特征层面决定而非标签层面。
原文摘要 · Abstract (English)
Grokking -- generalization long after training-set interpolation -- has been accelerated by structure-agnostic interventions (gradient filtering, weight-norm clamping, geometric penalties). Whether the delay specifically measures the time to form task-structured representations has remained observational. We test it causally by injecting representational priors of varying content into a one-layer transformer learning modular addition, via a supervised-contrastive loss whose positives encode (i) the task's true structure ($(a+b) \bmod p$), (ii) a coherent-but-wrong sibling ($(a-b) \bmod p$), or (iii) a random partition -- all with identical loss form, strength, class sizes, and geometry. Whether generalization occurs follows a clean gradation: true 22/30 runs, sibling (same periodic features, wrong combination) 14/15, random (only memorizable) 0/20 (Fisher $p=1.3\times10^{-7}$). A weight-norm-matched control replaying the norm trajectory onto plain cross-entropy generalizes 0/15, ruling out the norm as mediator. Probes show structure formation precedes and predicts generalization in all runs. Only the true structure also accelerates grokking (up to $2.75\times$), but this is dose-dependent and bimodal. We then confirm the mechanism by prediction: because the acceleration is gated by a weight-norm side-effect, clamping the norm during training yields a reliable, standalone accelerator with a median $8.6\times$ speedup (up to $22\times$ on the fastest seeds, under 1000 epochs), growing monotonically as the norm is held lower; the residual stalls also vanish, though significant only pooled over the two mitigations run at both strengths ($0/40$ vs $6/20$, $p=7.7\times10^{-4}$), not per method. The grokking delay is, causally, the time to form the right representational structure -- decided at the level of features, not labels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。