仅用数据统计就能预测核回归的学习曲线。
Predicting kernel regression learning curves from only raw data statistics
- 基于数据协方差和目标函数的多项式分解，构建学习曲线预测框架。
- 实验证明在真实图像数据上，该方法可准确预测测试风险随样本量的变化。
- 适用于研究模型性能与数据结构关系的研究者，尤其关注理论解释性。
我们研究了在真实数据集（包括CIFAR-5m、SVHN和ImageNet）上使用常见旋转不变核的核回归。提出一个理论框架，仅通过两个测量值——经验数据协方差矩阵和目标函数 $f_*$ 的经验多项式分解——即可预测学习曲线（测试风险随样本量变化）。核心创新是针对各向异性数据分布，对核的特征值和特征函数进行解析近似，该近似被称为赫米特特征结构假设（HEA），其特征函数类似于数据的赫米特多项式。我们在高斯数据上证明了HEA，但发现真实图像数据通常“足够接近高斯”，使HEA在实践中表现良好，从而能利用已有结果将核特征结构与测试风险关联。进一步地，在特征学习阶段，多层感知机（MLP）按HEA预测的顺序学习赫米特多项式。该框架为非平凡学习算法在真实数据上的端到端理论提供了概念验证，即从数据结构直接推导模型性能是可能的。
原文摘要 · Abstract (English)
We study kernel regression with common rotation-invariant kernels on real datasets including CIFAR-5m, SVHN, and ImageNet. We give a theoretical framework that predicts learning curves (test risk vs. sample size) from only two measurements: the empirical data covariance matrix and an empirical polynomial decomposition of the target function $f_*$. The key new idea is an analytical approximation of a kernel's eigenvalues and eigenfunctions with respect to an anisotropic data distribution. The eigenfunctions resemble Hermite polynomials of the data, so we call this approximation the Hermite eigenstructure ansatz (HEA). We prove the HEA for Gaussian data, but we find that real image data is often "Gaussian enough" for the HEA to hold well in practice, enabling us to predict learning curves by applying prior results relating kernel eigenstructure to test risk. Extending beyond kernel regression, we empirically find that MLPs in the feature-learning regime learn Hermite polynomials in the order predicted by the HEA. Our HEA framework is a proof of concept that an end-to-end theory of learning which maps dataset structure all the way to model performance is possible for nontrivial learning algorithms on real datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。