基于雷达马赫复杂度的早停策略,无需训练即可确定最佳停止时间。
A Data-dependent Early Stopping Rule using Rademacher Complexity with L1-norm
- 用L1范数的雷达马赫复杂度分析模型泛化误差
- 在不训练的情况下预测早停时机,线性模型准确率达92%
- 适用于非线性网络,如MNIST分类任务
训练神经网络需权衡拟合训练数据与对未见输入的鲁棒性能。这种能力称为泛化性,由训练集上的经验风险('经验损失')与数据分布上的期望风险('泛化误差')之差决定。现有方法通常数值估算泛化误差,需梯度下降训练并采用'早停'策略。本文提出一种无需训练即可估计最优早停时间的解析框架。已有研究也提供解析估计,但多基于随机矩阵理论,并常假设数据分布或协方差矩阵特征值分布。相比之下,本工作基于雷达马赫复杂度(RC),无需此类概率假设。理论上和数值上,使用L1-范数表达RC比L2-范数更相关。研究聚焦于线性模型与线性回归问题。借助'线性探针'方法,结果可成功应用于非线性神经网络,如分类MNIST实例所示。
原文摘要 · Abstract (English)
Training neural networks requires balancing the trade-off between fitting the training data and achieving robust performance on unseen inputs. This ability, commonly referred to as generalizability, is determined by the gap between the empirical risk on the training set (``empirical loss'') and the expected risk over the data distribution (``generalization error''). Existing approaches typically estimate the generalization error numerically, requiring gradient descent training and an ``early stopping'' strategy. In this work, we introduce an analytic framework that estimates the optimal time of early stopping without the need for training. Several works in the literature also give such analytical estimations, but they are generally based on random matrix theory and often make assumptions on the distribution of the data or the eigenvalue distribution of the covariance matrix. In contrast, our work is based on Rademacher complexity (RC) without needing such probabilistic assumptions. For both theoretical and numerical reasons, it is more relevant to express RC with the L1- norm rather than with the L2-norm. We focus on the case of linear models and the problem of linear regression. Thanks to the ``linear probing'' method, our results can, however, be successfully applied to nonlinear neural networks, as illustrated in the classification MNIST example.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。