arXiv:2605.12394cs.LGcs.AI2026-05被引 1

用随机矩阵理论检测神经网络过拟合,无需数据即可发现隐藏的过拟合信号。

Detecting overfitting in Neural Networks during long-horizon grokking using Random Matrix Theory

论文配图:Detecting overfitting in Neural Networks during long-horizon grokking using Random Matrix Theory
图 1 · 摘自论文原文
  • 通过重置权重并分析谱分布异常,识别出过拟合的结构性信号
  • 发现长期学习中存在测试准确率下降而训练准确率仍高的反向突现阶段
  • 适用于大模型过拟合监控,尤其适合研究基础模型的泛化风险

在不访问训练或测试数据的情况下,我们提出一种基于随机矩阵理论的新方法,用于检测深度学习模型中过拟合的开始。对每一层权重矩阵,我们逐元素随机化,将随机化后的经验谱分布拟合至马尔琴科-帕斯图分布,并识别违反自平均性的显著异常值,称为相关陷阱(Correlation Traps)。在长期学习中的过拟合初期——我们称之为“反向突现”阶段——这些陷阱出现并增多、放大,此时测试准确率下降而训练准确率保持高位。陷阱可能无害或损害泛化能力;我们通过在随机数据上运行模型并评估输出逻辑斯蒂的JS散度来区分其性质。结果表明,反向突现是具有高训练准确率和下降测试准确率的额外突现阶段,其结构不同于预突现阶段。更广泛地,我们发现部分基础规模大模型也表现出此类陷阱,暗示潜在有害过拟合的存在。

原文摘要 · Abstract (English)

Training Neural Networks (NNs) without overfitting is difficult; detecting that overfitting is difficult as well. We present a novel Random Matrix Theory method that detects the onset of overfitting in deep learning models without access to train or test data. For each model layer, we randomize each weight matrix element-wise, $\mathbf{W} \to \mathbf{W}^{\mathrm{rand}}$, fit the randomized empirical spectral distribution with a Marchenko-Pastur distribution, and identify large outliers that violate self-averaging. We call these outliers Correlation Traps. During the onset of overfitting, which we call the "anti-grokking" phase in long-horizon grokking, Correlation Traps form and grow in number and scale as test accuracy decreases while train accuracy remains high. Traps may be benign or may harm generalization; we provide an empirical approach to distinguish between them by passing random data through the trained model and evaluating the JS divergence of output logits. Our findings show that anti-grokking is an additional grokking phase with high train accuracy and decreasing test accuracy, structurally distinct from pre-grokking through its Correlation Traps. More broadly, we find that some foundation-scale LLMs exhibit the same Correlation Traps, indicating potentially harmful overfitting.

过拟合检测随机矩阵大模型分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。