无需模型结构,仅用单个数据集就能评估机器学习模型的泛化误差。
Perturbing the Derivative: Wild Refitting for Model-Free Evaluation of Machine Learning Models under Bregman Losses
- 通过扰动导数空间构造伪输出,再重新训练预测器来估计风险
- 在固定设计下以高概率保证上界,且不依赖函数类全局结构
- 适合评估深度网络等黑箱模型,理论严谨且实用性强
我们研究了在Bregman损失下经典正则化经验风险最小化(ERM)的过量风险评估问题。通过引入野生重拟合(wild refitting)思想,可仅利用单一数据集和训练过程的黑盒访问,通过随机Rademacher对称化及导数空间中的扰动构造人工输出,再重新训练第二个预测器,从而高效上界过量风险。该方法不依赖底层函数类的全局结构,具有内在的模型无关性。我们在固定设计设定下建立了高概率性能保证,证明在适当选择野生噪声尺度时,该方法能有效给出过量风险的上界。因此,该工作为评估现代复杂黑箱模型(如深度神经网络和生成模型)提供了理论可行的框架,尤其适用于传统学习理论难以处理的情形。
原文摘要 · Abstract (English)
We study the excess risk evaluation of classical penalized empirical risk minimization (ERM) with Bregman losses. We show that by leveraging the idea of wild refitting, one can efficiently upper bound the excess risk through the so-called "wild optimism," without relying on the global structure of the underlying function class. This property makes our approach inherently model-free. Unlike conventional analysis, our framework operates with just one dataset and black-box access to the training procedure. The method involves randomized Rademacher symmetrization and constructing artificially modified outputs by perturbation in the derivative space with appropriate scaling, upon which we retrain a second predictor for excess risk estimation. We establish high-probability performance guarantee under the fixed design setting, demonstrating that wild refitting under Bregman losses, with an appropriately chosen wild noise scale, yields a valid upper bound on the excess risk. Thus, our work is promising for theoretically evaluating modern opaque ML models, such as deep neural networks and generative models, where the function class is too complex for classical learning theory and empirical process techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。