通过严格数学证明,自蒸馏可显著提升岭回归性能,且支持负混合权重。
Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning
- 在无分布假设下,推导出最优混合权重的闭式解及其符号规律
- 理论证明在所有非平稳正则化点上,学生模型风险严格优于教师模型
- 提出无需调参或重训练的一次性调参方法,实测验证有效
自蒸馏(SD)是将学生模型用真实标签与教师预测的混合数据进行再训练的过程。本文研究了无约束条件下岭回归的自蒸馏,允许混合权重ξ超出[0,1]区间。在仅依赖训练数据、无分布假设的前提下,证明对于任意平方预测风险(包括分布外情形),只要教师模型的岭风险函数在正则化参数λ>0处非平稳(即导数R'(λ)≠0),最优混合的学生模型风险严格优于教师模型。我们给出了任意λ下的最优混合权重ξ⋆(λ)的闭式表达,并发现其遵循符号规则:sign(ξ⋆(λ)) = -sign(R'(λ)),意味着ξ⋆(λ)可为负值,出现在过正则化区域。进一步,在比例渐近(n,p→∞且p/n收敛)下,针对一般各向异性协方差和确定性信号,我们推导出最优自蒸馏风险的精确确定性等价物。该分析通过块线性化技术将标准二阶岭模型扩展至四阶,具有独立研究价值。从实用角度,我们提出一种一致的一次性调参方法,无需网格搜索、样本分割或重新拟合即可估计ξ⋆。真实数据集与预训练神经网络特征上的实验验证了理论结果及方法有效性。
原文摘要 · Abstract (English)
Self-distillation (SD) is the process of retraining a student on a mixture of ground-truth labels and the teacher's own predictions using the same architecture and training data. Although SD has been empirically shown to often improve generalization, its formal guarantees remain limited. We study SD for ridge regression in unconstrained setting in which the mixing weight $ξ$ may be outside the unit interval. Conditioned on the training data and without any distributional assumptions, we prove that for any squared prediction risk (including out-of-distribution), the optimally mixed student strictly improves upon the ridge teacher for every regularization level $λ> 0$ at which the teacher ridge risk $R(λ)$ is nonstationary (i.e., $R'(λ) \neq 0$). We obtain a closed-form expression for the optimal mixing weight $ξ^\star(λ)$ for any value of $λ$ and show that it obeys the sign rule: $\operatorname{sign}(ξ^\star(λ))=-\operatorname{sign}(R'(λ))$. In particular, $ξ^\star(λ)$ can be negative, which is the case in over-regularized regimes. To quantify the risk improvement due to SD, we derive exact deterministic equivalents for the optimal SD risk in the proportional asymptotics regime (where the sample and feature sizes $n$ and $p$ both diverge but their aspect ratio $p/n$ converges) under general anisotropic covariance and deterministic signals. Our asymptotic analysis extends standard second-order ridge deterministic equivalents to their fourth-order analogs using block linearization, which may be of independent interest. From a practical standpoint, we propose a consistent one-shot tuning method to estimate $ξ^\star$ without grid search, sample splitting, or refitting. Experiments on real-world datasets and pretrained neural network features support our theory and the one-shot tuning method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。