用几何距离比提前检测模型突然泛化,比传统方法早9%到73%。
ILDR: Geometric Early Detection of Grokking
- 计算最后一层特征的类间距与类内散度之比,捕捉表征空间重组。
- 在验证准确率提升前2.5倍基线值时即触发信号,领先950±250步。
- 适合研究泛化突变、优化训练终止或控制模型过渡的科研人员。
Grokking指神经网络在训练准确率已达100%后,验证准确率才突然提升的延迟泛化现象。现有检测方法间接且不稳定:权重范数滞后于转变,GrokFast的梯度指数移动平均在无梯度放大时跨种子波动大。本文提出类间/类内距离比(ILDR),基于倒数第二层特征计算类中心间距与类内散度之比。该几何指标在验证准确率变化前2.5倍基线值时上升并越过阈值,预示表征空间的早期重组。其理论基础为Fisher线性判别准则,计算复杂度O(|C|² + N),仅使用保留数据,避免记忆效应。在模运算和置换群组合(S5)任务中,ILDR领先训练预算的9%至73%,且任务代数复杂度越高,领先越显著。八组随机种子下,平均领先950±250步,变异系数26%,后突变阶段方差下降1696倍,符合表征空间的尖锐相变。以ILDR为停止条件可平均减少18.6%训练量。在阈值处干预优化器,能双向调控突变过程,表明其反映的是泛化背后的表征状态而非下游相关物。
原文摘要 · Abstract (English)
Grokking describes a delayed generalization phenomenon in which a neural network achieves perfect training accuracy long before validation accuracy improves, followed by an abrupt transition to strong generalization. Existing detection signals are indirect: weight norm reflects parameter-space regularization and consistently lags the transition, while GrokFast's slow gradient EMA, used without gradient amplification, is unstable across seeds with standard deviation exceeding mean lead time. We propose the Inter/Intra-class Distance Ratio (ILDR), a geometric metric computed on second-to-last layer representations as the ratio of inter-class centroid separation to intra-class scatter. ILDR provides an early detection signal: it rises and crosses a threshold at 2.5 times its baseline before the grokking transition appears in validation accuracy, indicating early geometric reorganization in representation space. Grounded in Fisher's linear discriminant criterion, ILDR requires no eigendecomposition and runs in O(|C|^2 + N). It is evaluated exclusively on held-out data, making it robust to memorization effects. Across modular arithmetic and permutation group composition (S5), ILDR leads the grokking transition by 9 to 73 percent of the training budget, with lead time increasing with task algebraic complexity. Over eight random seeds, ILDR leads by 950 +/- 250 steps with a coefficient of variation of 26 percent, and post-grokking variance drops by 1696 times, consistent with a sharp phase transition in representation space. Using ILDR as an early stopping trigger reduces training by 18.6 percent on average. Optimizer interventions triggered at the ILDR threshold demonstrate bidirectional control over the transition, suggesting ILDR tracks representational conditions underlying generalization rather than a downstream correlate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。