早停让过参数逻辑回归的梯度下降更准确，避免了过拟合。
Benefits of Early Stopping in Gradient Descent for Overparameterized Logistic Regression
- 用早停控制梯度下降，实现隐式正则化。
- 早停后风险趋于零，而长期迭代风险无穷大。
- 适合高维数据建模，尤其在样本不足时有效。
在高维过参数逻辑回归中，梯度下降（GD）迭代的范数发散但方向收敛到最大ℓ₂-边界解——即梯度下降的隐式偏差。本文研究了在模型正确设定下，早停带来的额外正则化效应。首先，我们证明：早停后的GD其过量逻辑损失趋于零，而收敛时的GD损失趋于无穷，表明早停后的方法是统计一致的，而渐近GD则不一致。其次，为获得小的零一损失，早停GD只需多项式量级样本，而任何插值估计器（包括渐近GD）需指数级样本。这一差距凸显了早停在过参数情形下的统计优势。最后，我们建立了早停GD与ℓ₂-正则化经验风险最小化器在范数和角度上的非渐近界，将梯度下降的隐式正则化与显式ℓ₂-正则化联系起来。
原文摘要 · Abstract (English)
In overparameterized logistic regression, gradient descent (GD) iterates diverge in norm while converging in direction to the maximum $\ell_2$-margin solution -- a phenomenon known as the implicit bias of GD. This work investigates additional regularization effects induced by early stopping in well-specified high-dimensional logistic regression. We first demonstrate that the excess logistic risk vanishes for early-stopped GD but diverges to infinity for GD iterates at convergence. This suggests that early-stopped GD is well-calibrated, whereas asymptotic GD is statistically inconsistent. Second, we show that to attain a small excess zero-one risk, polynomially many samples are sufficient for early-stopped GD, while exponentially many samples are necessary for any interpolating estimator, including asymptotic GD. This separation underscores the statistical benefits of early stopping in the overparameterized regime. Finally, we establish nonasymptotic bounds on the norm and angular differences between early-stopped GD and $\ell_2$-regularized empirical risk minimizer, thereby connecting the implicit regularization of GD with explicit $\ell_2$-regularization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。