arXiv:2605.08237cs.LGstat.ML2026-05被引 2

通过分布谱诊断,提前定位模型的‘顿悟’现象。

Distributional Spectral Diagnostics for Localizing Grokking Transitions

论文配图:Distributional Spectral Diagnostics for Localizing Grokking Transitions
图 1 · 摘自论文原文
  • 用Wasserstein坐标和动态模分解分析训练轨迹分布变化
  • 残差指标在运行级别区分顿悟与非顿悟,AUROC达0.93
  • 可提前预警,适合研究神经网络泛化跃迁的学者

在‘顿悟’现象中,模型先拟合训练数据但测试准确率仍低,随后才开始泛化。我们探究能否从训练轨迹中提前定位这一转变,并将该问题形式化为具有明确阈值、误报率与提前量权衡的诊断任务。针对特定任务的可观测变量被归纳为经验分布,映射至Wasserstein/分位数坐标,并通过Hankel动态模分解(DMD)分析;生成的重构残差、谱特征与有效秩构成诊断输出。在保留的模加法Transformer实验中,残差在运行级别对顿悟与非顿悟的判别达到约0.93的AUROC;在固定持续阈值规则下,真阳性警报可提前于转折点出现,且同时报告误报率与不确定性区间。扰动实验表明,在测试的wd=1池中,高残差窗口的短时扰动偏差约为低残差窗口的3倍。在同数据范数窗口对照中,扰动敏感性与残差排序一致,而非总参数范数排序,说明残差并非单纯总范数代理。范数信号仍是强运行级状态指示器,对数概率在所试可观测物中表现最佳。我们将残差定位为在模运算Transformer设置下的窗口级监控与定位信号,而非通用早期预警器或干预规则。

原文摘要 · Abstract (English)

In grokking, a model first fits the training data while test accuracy remains low, and only later begins to generalize. We ask whether this transition can be localized from observed training trajectories before the test accuracy rises, and formulate grokking transition localization as a diagnostic problem with an explicit threshold/FPR/lead-time trade-off. Task-dependent observables are summarized as empirical distributions, mapped to Wasserstein/quantile coordinates, and analyzed by Hankel dynamic mode decomposition (DMD); the resulting reconstruction residual, together with spectrum and effective rank, forms the diagnostic output. On held-out modular-addition Transformer runs, the residual achieves AUROC \(\approx \) 0.93 for grokking-vs-non-grokking discrimination at the run level; under a fixed sustained-threshold operating rule, true-positive alarms can precede onset, with lead time reported jointly with false-alarm rate and uncertainty intervals. Perturbation experiments show that, in the tested \(wd=1\) pool, high-residual windows exhibit about \(3\times\) larger short-horizon perturbation deviation than low-residual windows. In a same-data norm-window control, perturbation sensitivity aligns with the residual ordering rather than total-parameter-norm ordering, suggesting that the residual is not merely a total-norm proxy at the window level in the studied \(wd=1\) dynamics. Norm signals remain strong run-level regime indicators, and log-probability performs best among the observables tested under the current protocol. We position the residual as a window-level monitoring and localization signal in the studied modular-arithmetic Transformer settings, not a universal early-warning predictor or an intervention rule.

顿悟现象动态模分解分布诊断模型监控

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。