提出一种简单高斯近似,精准刻画岭回归在小样本下的偏差-方差权衡。
A Simple Approximation to the Distribution of the Ridge Regression Estimator
- 基于非标准渐近理论,将正则化参数与样本量同比增大
- 在低维模型下允许异方差和自相关,逼近精度更高
- 新选参策略可最小化平均或最坏情况的预测风险
我们提出了对经典岭回归估计量有限样本分布的一种简单高斯近似。该近似捕捉了在小样本下,岭回归估计量通过权衡偏差与方差来降低估计和预测误差的本质特性。近似基于非标准渐近框架:一是令估计量的正则化参数随样本量同比例增长;二是将总体回归系数视为相对于定义收缩方向的参考向量的局部扰动。相比文献中其他渐近近似,我们的方法允许数据生成过程存在一般形式的异方差性和自相关性(代价是仅考虑低维模型,即协变量数量不随样本量增长)。利用该高斯近似,我们提出了两种新的正则化参数选择策略:分别以最小化平均或最坏情况下的超额预测风险为目标,风险计算基于所提出的高斯近似。
原文摘要 · Abstract (English)
We present a simple Gaussian approximation to the finite-sample distribution of the classical ridge regression estimator. Our approximation captures the fact that, in finite samples, the ridge regression estimator trades off bias and variance to reduce estimation and prediction error. Our approximation is based on nonstandard asymptotics where $i)$ we let the estimator's regularization parameter grow proportionally to the sample size; and $ii)$ we treat the population regression coefficients as \emph{local} to the reference vector that defines the estimator's direction of shrinkage. In contrast to other asymptotic approximations in the literature, we allow for general forms of heteroskedasticity and autocorrelation in the data generating process (at the cost of considering a low-dimensional model where the number of covariates is not allowed to grow with the sample size). We use our simple Gaussian approximation to propose two new strategies to select the regularization parameter for the ridge regression estimator. The suggested strategies select the regularization parameter to minimize either average or worst-case excess prediction risk, where risk is computed using our suggested Gaussian approximation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。