无需原始数据,仅用预测结果就能提升模型性能。
Prediction-Only Distillation in Linear and Logistic Regression
- 用教师模型伪标签新数据训练学生模型,再混合两者预测。
- 混合后风险严格低于教师模型,即使数据分布不同也有效。
- 只需少量标注数据校准,无需重新训练即可确定最优混合权重。
自蒸馏(SD)通常在可访问原始训练数据时研究,但在实际应用中,原始标签数据常不可得,仅能获取已训练的预测器和新的无标签特征。本文提出一种预测-混合方案:纯蒸馏学生模型基于教师模型伪标签的新特征进行训练,最终预测为教师与学生预测的仿射组合。在比例渐近下,针对岭回归,我们推导出最优混合预测风险的确定性等价表达式,适用于一般各向异性协方差和确定性信号。结果显示,该风险在几乎所有教师与学生正则化参数对下均严格小于教师风险,包括新特征分布偏离原始数据或协方差各向同性的情况。进一步证明,最优混合权重无法仅从无标签数据中识别,但可通过一个独立的小规模标注校准集,在单次后处理步骤中一致估计,无需额外模型拟合。对于二分类逻辑回归,预测混合亦优于教师模型和纯蒸馏分类器。
原文摘要 · Abstract (English)
Self-distillation (SD) is typically studied when the student is retrained on the teacher's original training inputs. In many practical deployments, however, the labeled training data are no longer available, and one has access only to the trained predictor and fresh unlabeled covariates. We study SD in this prediction-only regime through a fresh-X prediction-mixed scheme: a pure-distilled student is trained on fresh covariates pseudo-labeled by the teacher, and the final predictor is an affine combination of the teacher and student predictions. For ridge regression under proportional asymptotics, we derive deterministic equivalents for the optimally mixed prediction risk under general anisotropic covariance and deterministic signal. We show that this risk is strictly smaller than the teacher risk for almost every pair of teacher and student regularization levels, including when the fresh covariates are out-of-distribution and even when their covariance is isotropic. We further show that the optimal mixing weight cannot be identified from unlabeled data alone, but can be consistently estimated in a single post-training step using a small independent labeled calibration set, without additional model fitting. Finally, for binary logistic regression, we show that prediction mixing can outperform both the teacher and the pure-distilled classifier.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。