自训练在高维线性回归中存在去噪与信号遗忘的权衡,导致测试误差呈U形变化。
Why Self-Training Helps and Hurts: Denoising vs. Signal Forgetting
- 通过递推公式分析自训练中信号遗忘与数据投影去噪的动态平衡
- 高维下测试风险先降后升,存在最优早停时机
- 提出广义交叉验证法实现全数据驱动的停止时间选择
在过参数化线性回归设置中,研究迭代自训练(自蒸馏)过程:初始模型在含噪声标签上训练,后续每轮在新协变量上使用前一轮模型生成的无噪声伪标签进行再训练。在高维情形下,推导出预测风险与有效噪声的确定性等价递推关系,并证明经验量在这些极限附近高度集中。该递推揭示两种竞争机制:随迭代增长的系统性信号遗忘,以及通过数据相关投影实现的随机性去噪。二者相互作用产生U形测试风险曲线和最优早停时间。在带状协方差模型中,迭代行为表现为依赖迭代次数的谱滤波器,保留强主成分方向而抑制弱方向,形成区别于岭回归的隐式软特征选择。最后提出一种迭代广义交叉验证准则,并证明其对自训练轨迹上风险估计具有统一一致性,支持完全数据驱动的停止时间选取。合成协方差上的实验验证了理论并展示了预测中的去噪-遗忘权衡。
原文摘要 · Abstract (English)
Iterative self-training (self-distillation) repeatedly refits a model on pseudo-labels generated by its own predictions. We study this procedure in overparameterized linear regression: an initial estimator is trained on noisy labels, and each subsequent iterate is trained on fresh covariates with noiseless pseudo-labels from the previous model. In the high-dimensional regime, we derive deterministic-equivalent recursions for the prediction risk and effective noise across iterations, and prove that the empirical quantities concentrate sharply around these limits. The recursion separates two competing forces: a systematic component that grows with iteration due to progressive signal forgetting, and a stochastic component that decays due to denoising via repeated data-dependent projections. Their interaction yields a $U$-shaped test-risk curve and an optimal early-stopping time. In spiked covariance models, iteration further acts as an iteration-dependent spectral filter that preserves strong eigendirections while suppressing weaker ones, inducing an implicit form of soft feature selection distinct from ridge regression. Finally, we propose an iterated generalized cross-validation criterion and prove its uniform consistency for estimating the risk along the self-training trajectory, enabling fully data-driven selection of the stopping time and regularization. Experiments on synthetic covariances validate the theory and illustrate the predicted denoising-forgetting trade-off.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。