arXiv:2607.12735cs.LG2026-07

发现让模型快速泛化的关键:早期使用无标签不变性先验可大幅提速。

What Makes a Representational Prior Work? Feature Families, Label-Free Invariances, and Critical Windows in Grokking

  • 用无标签的对称性先验替代有监督信号,仍能加速泛化。
  • 仅在前2000轮应用先验,效果媲美全程使用,速度提升2.7倍。
  • 特征家族匹配是泛化前提,适合研究模型学习机制的人。

前期研究指出,模型‘突现’(grokking)的延迟源于任务结构表征的形成时间,可通过对比先验注入。本文在188次新实验中,从四个维度分析先验有效性的原因:特征家族不匹配(如幅度带)的先验会阻断泛化(1/15成功,而随机划分0/20,p=0.43),证实先验作用于电路级特征;完全无标签的对称性先验(仅$(a,b) ightarrow(b,a)$)在15/15次实验中实现中位2.7倍加速,优于有标签先验(p=0.038),结合权重范数约束后达到最强效果(中位17倍,5/5);仅需在前2000轮(总预算4%)应用先验即可实现10/10成功,速度达2.7倍,优于持续使用(8/10,1.25倍)或延迟窗口(2.1倍);该现象在模乘法、不同深度与归一化方式下复现,且范数约束扫面验证核心结论:结构注入使权重范数延迟指数下降约17倍(普通交叉熵每+10范数单位慢31倍,下界,更高值被截断;加先验后仅慢1.22倍)。无法控制的提前泛化任务无延迟可调。

原文摘要 · Abstract (English)

Companion work showed the grokking delay is causally the time to form task-structured representations, injectable via a contrastive prior. Here we characterize what makes such a prior work, across four axes, in 188 new runs. Content: a coherent, learnable prior built from the wrong feature family (magnitude bands) blocks generalization like a random partition (1/15 vs 0/20 grok; $p=0.43$ between them), confirming the companion's prediction that priors act at the level of the circuit's features. Supervision: a fully label-free invariance prior -- positives are commuted pairs $(a,b)\sim(b,a)$ only -- generalizes in 15/15 runs at a median $2.7\times$ speedup, more reliably than the label-supervised prior itself ($p=0.038$), and combined with a weight-norm clamp yields the strongest method we test (median $17\times$, 5/5) -- strongest meaning reliably fast: plain cross-entropy with a clamp matches this speed only at the exact critical norm, while the prior keeps it fast across the entire clamp range. Timing: the prior is only needed early -- applied solely during the first 2000 epochs (4% of budget) it generalizes 10/10 at $2.7\times$, beating continuous application (8/10, $1.25\times$) and a duration-matched later window ($2.1\times$). Setting: the dissociation replicates on modular multiplication and across depths and normalization variants, and a clamp sweep quantifies the companion's central claim: structure injection flattens the weight-norm delay-law exponent about 17-fold (plain cross-entropy slows $31\times$ per +10 norm units, a lower bound as higher cells are censored, versus $1.22\times$ with the prior). Honest boundary: tasks that generalize before memorizing have no delay to control. Feature-family alignment decides whether a prior permits generalization; invariance content suffices for acceleration without labels; a brief early window captures nearly all of the benefit.

模型泛化无监督学习先验机制学习加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。