用教师模型预测学生误差，减少偏差传播，提升估计精度。
Residual-as-Teacher: Mitigating Bias Propagation in Student--Teacher Estimation
- 以教师模型指导学生预测残差，而非直接模仿输出
- 理论证明可降低教师偏差影响，实现最优预测率
- 适合存在标签偏差或数据偏移的场景，如图像分类
我们研究学生-教师框架下的统计估计问题，其中预训练教师模型的输出用于指导学生模型。标准方法是让学生直接匹配教师输出（软匹配，SM），但会直接传播教师中的系统性偏差或建模错误，导致学生性能下降。本文提出并分析一种新方案——残差作为教师（RaT），即利用教师估计学生预测的残差。理论分析表明，该方法等价于求解一个理想优化问题的近端梯度算法，能有效缓解教师偏差。对于一般的学生-教师对，我们给出了任意RaT固定点的非渐近超额风险界及迭代收敛性保证；对于基于核函数的学生-教师对，证明了显著分离：RaT达到最小最大最优率，而SM在任何样本量下均存在常数级预测误差。合成数据与ImageNette分类任务在协变量偏移下的实验验证了理论结果。
原文摘要 · Abstract (English)
We study statistical estimation in a student--teacher setting, where predictions from a pre-trained teacher are used to guide a student model. A standard approach is to train the student to directly match the teacher's outputs, which we refer to as student soft matching (SM). This approach directly propagates any systematic bias or mis-specification present in the teacher, thereby degrading the student's predictions. We propose and analyze an alternative scheme, known as residual-as-teacher (RaT), in which the teacher is used to estimate residuals in the student's predictions. Our analysis shows how the student can thereby emulate a proximal gradient scheme for solving an oracle optimization problem, and this provably reduces the effect of teacher bias. For general student--teacher pairs, we establish non-asymptotic excess risk bounds for any RaT fixed point, along with convergence guarantees for the student-teacher iterative scheme. For kernel-based student--teacher pairs, we prove a sharp separation: the RaT method achieves the minimax-optimal rate, while the SM method incurs constant prediction error for any sample size. Experiments on both synthetic data and ImageNette classification under covariate shift corroborate our theoretical findings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。