arXiv:2505.24313cs.LG2025-05中稿 · ICML被引 4

弱教师生成标签,强学生模型反而超越,理论揭示其机制。

Weak-to-Strong Generalization via Bregman Bias-Variance Decomposition

  • 用Bregman散度分解偏差-方差,突破传统凸假设限制。
  • 学生逼近教师后验均值时,弱到强泛化更可能发生。
  • 反向交叉熵能提升性能,适合模型蒸馏与小样本学习。

弱到强泛化(W2SG)指一个强大学生模型在弱教师生成标签上训练后,最终优于教师。本文基于Bregman散度的广义偏差-方差分解,理论上揭示了这一现象的成因。我们发现学生与教师的期望种群风险差距由两者间期望误配决定,且无需假设学生假设类为凸集。结果表明,当学生有效逼近教师后验均值时,W2SG更易出现。针对平方损失,给出学生收敛至后验均值教师的充分条件;增大模型规模可确保该收敛。对于交叉熵损失,分析表明降低学生预测分布熵有助于促进W2SG。此外,反向交叉熵相比标准前向交叉熵对教师预测不确定性不敏感。实验验证了这些理论见解,并证明引入反向交叉熵可稳定提升学生性能。

原文摘要 · Abstract (English)

Weak-to-strong generalization (W2SG) is the phenomenon in which a powerful student model, trained on labels produced by a weaker teacher, ultimately outperforms the teacher on the target task. In this work, we theoretically investigate how W2SG can arise via a generalized bias-variance decomposition under Bregman divergence. We show that the expected population risk gap between the student and the teacher is characterized by the expected misfit between the two models. Unlike earlier misfit-based analyses, our theory removes several restrictive assumptions, e.g., it does not require the student hypothesis class to be convex. Our results indicate that W2SG is more likely when the student effectively approximates the teacher's posterior mean. Specializing to squared loss, we provide a sufficient condition (illustrated through a concrete example) under which the student converges to its posterior mean teacher; in particular, increasing the student model size can ensure this convergence. For cross-entropy loss, our analysis further suggests that lowering the entropy of the student's predictive distribution can promote W2SG. We also find that the reverse cross-entropy, unlike the standard forward cross-entropy, is less sensitive to the teacher's predictive uncertainty. Finally, we verify these theoretical insights empirically and demonstrate that incorporating reverse cross-entropy consistently improves student performance.

模型蒸馏泛化理论贝叶斯学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。