arXiv:2607.05104cs.LGcs.AI2026-07

在1.2万参数模型上发现,模型泛化依赖训练覆盖度,且极易受计算环境影响。

Grokking Is Conditional and Fragile: A Fully-Tractable, Multi-Seed Study at 12K Parameters

  • 用可完全解析的小模型研究泛化延迟现象,避免单次实验偏差
  • 泛化仅在训练数据覆盖度超过模数阈值时出现,与任务结构无关
  • 同一种子结果可能因CPU线程或硬件差异而反转,说明现象极脆弱

Grokking——即网络在完全拟合训练集后长时间才实现泛化的现象——通常在过大模型中研究,且基于单次训练运行。本文改用一个约11,856参数的Llama风格Transformer(Glimmer-1-Base),在模运算任务上进行研究,模型小到可完全枚举权重、注意力机制及输入输出映射。通过多种子率测量而非单一结果,揭示grokking是一种条件性、脆弱的相变。其发生由训练集覆盖度决定,阈值更接近输出基数(模数)而非任务结构;该规律在相变后及十倍域大小变化下均成立。权重衰减在12K参数下重现Omnigrok的倒U型曲线,验证了率测量的有效性。此外,grokking处于数值敏感的临界状态:浮点环境微调——如CPU线程数(减少顺序)和CPU/GPU执行——均能翻转少数同种子的结果,但总体率无明显变化。将模型分解为子任务专家主要通过降低覆盖成本来提升性能,而非依赖监督信号。方法论上,固定数值环境下多种子控制推翻了三项基于单次运行的错误叙事,皆源于种子混淆。因此,grokking的证据单位应为固定数值环境下多种子率,并尽可能与模型直接读取结果交叉验证。

原文摘要 · Abstract (English)

Grokking -- the delayed onset of generalization long after a network has fit its training set - -is usually studied in models too large to read completely and reported from single training runs. We instead study a publicly released ~11,856-parameter Llama-style transformer (Glimmer-1-Base) on modular arithmetic, small enough to enumerate its weights, attention, and full input-output map, and we measure grokking as a multi-seed rate rather than a single outcome. In this fully-tractable regime grokking is a conditional, fragile phase transition. It is gated by training-set coverage, whose threshold tracks output cardinality (the modulus) more than task structure, an ordering that holds above the transition and across a ten-fold change in domain size. Weight decay reproduces the Omnigrok inverted-U at 12K parameters, a positive control on the rate measurement. Grokking also sits on a numerical knife-edge: two perturbations of the floating-point environment -- CPU thread count (reduction order) and CPU-versus-GPU execution -- each flip a minority of same-seed outcomes without a detectable shift in the aggregate rate. Decomposition into sub-task specialists helps chiefly by making coverage cheap rather than by adding supervision. Methodologically, multi-seed control under a fixed numerical environment overturns three dramatic single-run narratives in our own data, each a seed confound. The unit of evidence for grokking must therefore be a multi-seed rate under a pinned numerical environment, checked where possible against a direct reading of the model.

深度学习泛化可解释性模型行为

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。