提出用缩放乐观度衡量模型复杂度，揭示神经网络与核模型的本质差异。
Asymptotic Optimism of Random-Design Linear and Kernel Regression Models
- 基于随机设计推导线性与核回归的渐近乐观度公式
- 发现带ReLU的三层网络在乐观度上显著区别于核模型
- 提供真实数据下计算乐观度的重采样方法，适合模型评估研究者
我们推导了随机设计下线性回归模型的闭式渐近乐观度，并将其推广至核岭回归。以缩放后的渐近乐观度作为通用预测模型复杂度度量，研究了线性回归、正切核（NTK）回归和三层全连接神经网络的根本行为差异。贡献在于：为使用缩放乐观度作为模型预测复杂度度量提供了理论基础；并实证表明，带有ReLU的神经网络在此度量下表现出与核模型不同的行为。通过重采样技术，也可在真实数据上计算回归模型的乐观度。
原文摘要 · Abstract (English)
We derived the closed-form asymptotic optimism of linear regression models under random designs, and generalizes it to kernel ridge regression. Using scaled asymptotic optimism as a generic predictive model complexity measure, we studied the fundamental different behaviors of linear regression model, tangent kernel (NTK) regression model and three-layer fully connected neural networks (NN). Our contribution is two-fold: we provided theoretical ground for using scaled optimism as a model predictive complexity measure; and we show empirically that NN with ReLUs behaves differently from kernel models under this measure. With resampling techniques, we can also compute the optimism for regression models with real data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。