提出一种可计算且严谨的模型复杂度度量方法
A Rigorous, Tractable Measure of Model Complexity
- 基于模型梯度在不同输入下的相似性定义复杂度
- 能统一解释多项式、核函数、决策树等模型的复杂度
- 适用于各类模型，助力理解过拟合与双下降现象
准确评估模型复杂度对模型解释、泛化能力和模型选择至关重要。然而，现有复杂度度量大多依赖启发式假设或计算成本过高。本文提出一种数学严谨且易于计算的模型复杂度度量，基于模型梯度在不同输入间的相似性，适用于任意参数化模型及基于核的非参数模型。我们证明该度量可统一涵盖多项式回归的次数、Matérn核的长度尺度、k近邻的邻居数、决策树的分裂数和随机森林的树数量等特定模型的复杂度度量。此外，该度量为随机傅里叶特征、随机森林、神经网络和梯度提升模型中的双下降现象提供了新见解。
原文摘要 · Abstract (English)
An accurate assessment of a model's complexity is crucial for topics such as interpretation, generalization, and model selection. However, most existing complexity measures either rely on heuristic assumptions or are computationally prohibitive. In this paper, we present a mathematically rigorous yet easy-to-compute measure of model complexity that is based on the similarities between the model gradients across inputs. It is thus well-defined for any parametric model, but also for kernel-based non-parametric models. We prove that our measure of complexity generalizes model-specific complexity measures such as polynomial degree (for polynomial regression), kernel length scale (for Matérn kernels), number of neighbors (for k-nearest neighbors), number of splits (for decision trees), and number of trees (for random forests). We also use our measure to obtain new insights into the double descent phenomenon for random Fourier features, random forests, neural networks, and gradient boosting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。