发现模型泛化后仍持续压缩表征，现有度量会高估收敛状态。
At-Grok Is Not Converged:A Measurement-Validity Audit for Grokking Representation Metrics
- 区分泛化起点与压缩过程，揭示延迟现象
- MLP中有效秩被高估3-5倍，Transformer高估1.3-1.5倍
- 适用于检测表征度量偏差，适合模型分析研究者
在模运算任务中，网络的嵌入表示在泛化完成后仍持续压缩，长达数万步。在MLP上，于泛化跃迁点读取的有效秩比收敛值高3-5倍；在训练至收敛的Transformer上则高1.3-1.5倍。此外，该度量还掩盖了哪些单元发生压缩的事实。压缩滞后于准确率跃迁，时间延迟至少为10,000步，而非同步。单变量消融显示：向同构Transformer加入LayerNorm后，泛化步骤完成的压缩比例从0.87降至0.25；预注册对照实验排除了尺度不变性作为机制。本文提出一套审计工具，可分离泛化起始与压缩过程，识别数据屏蔽、排除未完全泛化的边界单元，并验证参考基线已平稳。该工具集成功检测出自身分支中的虚假自信缺陷。二次分析发现，针对MLP的深度规律在Transformer上失效且符号反转，说明其缺乏普适性。代码与工具包已公开。
原文摘要 · Abstract (English)
On modular arithmetic, a network's embedding keeps compressing for tens of thousands of steps after it has already generalized. Reading effective rank at the grokking transition overstates the converged value by 3-5x on an MLP, and by 1.3-1.5x on a transformer trained to convergence; on the MLP it also erases which cells compress at all. Compression lags the accuracy transition by an amount on the order of the time-to-grok, at least 10,000 steps, rather than coinciding with it. A one-variable ablation shows what sets the lag size: adding LayerNorm to an otherwise identical transformer moves the fraction of compression done by the grok step from 0.87 to 0.25, and a pre-registered control rules out scale invariance as the mechanism. We package this as an audit that separates onset from compression, flags censoring, excludes boundary cells that never fully generalize, and checks that the reference floor has plateaued, with an adversarial suite that caught a false-confidence bug in our own branch. A secondary, MLP-specific depth law linking norm budget to converged floor fails a generality test on a transformer and flips sign under free weight decay. Code and the toolkit are released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。