arXiv:2506.04434cs.LGcs.AI2025-06被引 9

发现神经网络训练后期存在测试精度骤降的反生长现象,由权重矩阵异常引发。

Grokking and Generalization Collapse: Insights from \texttt{HTSR} theory

  • 用重尾自正则化理论分析三层MLP训练过程
  • 发现训练700万步后出现测试精度崩溃的反生长阶段
  • α值低于2且出现相关陷阱是过拟合的关键信号

我们使用3层MLP在MNIST的1000样本子集上训练,结合与不结合权重衰减,研究神经网络中的经典生长现象。通过训练长达10^7步,发现了全新的第三阶段——反生长:测试精度突然下降而训练精度保持完美。该现象与已知的预生长阶段不同,且无法被现有生长进度度量检测到。借助开源WeightWatcher工具,基于重尾自正则化(HTSR)理论,我们发现仅用层质量指标α即可明确划分三个阶段,而其他最优指标仅能识别前两个。反生长现象始终伴随α < 2及‘相关陷阱’——即随机化层权重矩阵中出现的异常奇异值,表明权重矩阵偏离典型分布,反映对训练集的过拟合。这些陷阱通过层间经验谱密度可视化和柯尔莫哥洛夫-斯米尔诺夫检验验证。激活稀疏性、绝对权重熵、电路复杂度、l²权重范数等对比度量可追踪预生长与生长,但无法区分生长与反生长。这一发现为无需测试数据即可测量过拟合与泛化崩溃提供了新路径,进一步支持了HTSR中α ≈ 2为普适层收敛目标的观点,并凸显α作为泛化度量的价值。

原文摘要 · Abstract (English)

We study the well-known grokking phenomena in neural networks (NNs) using a 3-layer MLP trained on 1 k-sample subset of MNIST, with and without weight decay, and discover a novel third phase -- \emph{anti-grokking} -- that occurs very late in training and resembles but is distinct from the familiar \emph{pre-grokking} phases: test accuracy collapses while training accuracy stays perfect. This late-stage collapse is distinct, from the known pre-grokking and grokking phases, and is not detected by other proposed grokking progress measures. Leveraging Heavy-Tailed Self-Regularization HTSR through the open-source WeightWatcher tool, we show that the HTSR layer quality metric $α$ alone delineates all three phases, whereas the best competing metrics detect only the first two. The \emph{anti-grokking} is revealed by training for $10^7$ and is invariably heralded by $α< 2$ and the appearance of \emph{Correlation Traps} -- outlier singular values in the randomized layer weight matrices that make the layer weight matrix atypical and signal overfitting of the training set. Such traps are verified by visual inspection of the layer-wise empirical spectral densities, and by using Kolmogorov--Smirnov tests on randomized spectra. Comparative metrics, including activation sparsity, absolute weight entropy, circuit complexity, and $l^2$ weight norms track pre-grokking and grokking but fail to distinguish grokking from anti-grokking. This discovery provides a way to measure overfitting and generalization collapse without direct access to the test data. These results strengthen the claim that the \emph{HTSR} $α$ provides universal layer-convergence target at $α\approx 2$ and underscore the value of using the HTSR alpha $(α)$ metric as a measure of generalization.

神经网络过拟合泛化能力生长现象

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。