arXiv:2602.12680stat.MLcs.LG2026-02

提出线性插值器的新权衡机制,解析过参数模型的性能差异。

A Regularization-Sharpness Tradeoff for Linear Interpolators

  • 用正则化与几何尖锐度分解选择惩罚项
  • 理论证明p≥2时ℓ^p正则器的通用表达式
  • 实验证明该权衡能区分强弱插值器

经典机器学习中偏差-方差权衡规则在过参数化场景下失效,表现为双下降曲线。最小范数插值估计器表现良好,提示需建立新权衡机制。本文为带ℓ^p正则的过参数化线性回归提出正则化-尖锐度权衡框架,受插值信息准则启发,将选择惩罚分解为正则项(衡量正则器与插值器对齐程度)和几何尖锐项(衡量局部扰动影响),形成类似偏差-方差的权衡。基于已有针对岭回归的分析,本文首次给出p≥2时ℓ^p正则器的通用表达式,并进一步扩展至ℓ^1正则的LASSO插值器,实现更强稀疏性。在含随机傅里叶特征与多项式的实际数据集上进行实证,验证了理论的有效性,表明该权衡可有效区分性能优异与较差的线性插值器。

原文摘要 · Abstract (English)

The rule of thumb regarding the relationship between the bias-variance tradeoff and model size plays a key role in classical machine learning, but is now well-known to break down in the overparameterized setting as per the double descent curve. In particular, minimum-norm interpolating estimators can perform well, suggesting the need for new tradeoff in these settings. Accordingly, we propose a regularization-sharpness tradeoff for overparameterized linear regression with an $\ell^p$ penalty. Inspired by the interpolating information criterion, our framework decomposes the selection penalty into a regularization term (quantifying the alignment of the regularizer and the interpolator) and a geometric sharpness term on the interpolating manifold (quantifying the effect of local perturbations), yielding a tradeoff analogous to bias-variance. Building on prior analyses that established this information criterion for ridge regularizers, this work first provides a general expression of the interpolating information criterion for $\ell^p$ regularizers where $p \ge 2$. Subsequently, we extend this to the LASSO interpolator with $\ell^1$ regularizer, which induces stronger sparsity. Empirical results on real-world datasets with random Fourier features and polynomials validate our theory, demonstrating how the tradeoff terms can distinguish performant linear interpolators from weaker ones.

过参数化正则化线性回归权衡机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。