揭示了高维数据下吉布斯采样泛化能力的内在机制
Generalization of Gibbs and Langevin Monte Carlo Algorithms in the Interpolation Regime
- 基于数据依赖的误差界分析，刻画低温度时泛化性能
- 训练误差小即预示低温度下泛化良好，即使标签随机也成立
- 算法可生成真实标签与随机标签的测试误差上界
本文在过参数化插值区间中，为吉布斯算法的期望误差提供了数据依赖性界。结果表明，在低温度区，泛化能力已由高温区较高的噪声训练误差所预示。这些界在用朗之万蒙特卡洛算法近似时仍保持稳定。该分析启发了一种计算边界的新算法：在MNIST、CIFAR-10和SVHN数据集上，对真实标签数据能给出接近实际测试误差的非平凡预测，而对随机标签数据仍能维持正确的测试误差上界。
原文摘要 · Abstract (English)
This paper provides data-dependent bounds on the expected error of the Gibbs algorithm in the overparameterized interpolation regime, where low training errors are also obtained for impossible data, such as random labels in classification. The results show that generalization in the low-temperature regime is already signaled by small training errors in the noisier high-temperature regime. The bounds are stable under approximation with Langevin Monte Carlo algorithms. The analysis motivates the design of an algorithm to compute bounds, which on the MNIST, CIFAR-10, and SVHN datasets yield nontrivial, close predictions on the test error for true labeled data, while maintaining a correct upper bound on the test error for random labels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。