arXiv:2608.04794cs.AIcs.LG2026-08被引 1

自蒸馏在难题上失效,因教师偏见导致学生学错方向。

Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation

论文配图:Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation
图 1 · 摘自论文原文
  • 教师依赖参考答案生成每字监督信号,但只学特定解法路径。
  • 学生损失持续下降,准确率不升反降,尤其在复杂任务中。
  • 适合研究训练机制偏差、模型推理能力评估的读者。

自蒸馏(SD)作为强化学习的高效替代,通过一个基于特权信息(PI)的自教师,为从未见过该信息的学生提供逐词监督。尽管已有报告称其在简单任务中有效,我们复现发现:在困难任务上,无论问答、数学、编码或多轮代理工具使用,不同推理模式、模型规模和PI形式下,模型的逐词损失持续下降,但验证准确率未提升,反而普遍下降。原因在于教师受特定参考解法影响产生偏见,使学生目标偏离正确性本身,转而惩罚推理中的犹豫表达,最终生成缺乏决断力的模型。这表明,仅靠自蒸馏无法保证任务成功,其优化信号与实际表现已脱钩。

原文摘要 · Abstract (English)

Self-distillation (SD) has emerged as a compute-efficient alternative to reinforcement learning with verifiable rewards: a self-teacher, conditioned on privileged information (PI) about the answer such as a reference solution, supplies dense per-token supervision to a student that never sees it. Reported gains, however, come almost exclusively from narrow, low-difficulty settings, leaving open a basic question: as a lone objective, with no reward term, does SD teach anything? We reproduce SDPO's reported gains in its easy setting, then apply the identical setup to difficult tasks and find that it does not. Across question answering, mathematics, coding, and multi-turn agentic tool use, across reasoning modes, model sizes, and forms of PI, and under both the SDPO and OPSD recipes, the per-token loss falls steadily while validation accuracy does not improve and typically degrades. We explain this failure through a single causal chain from the loss to the model it produces. The chain begins with PI bias: having seen one particular reference solution, the teacher's per-token target is pulled toward that trajectory rather than toward correctness in general, an effect we quantify with a PI Bias Score. Trained to match this target everywhere, the student's objective becomes nearly blind to whether a rollout is correct, and the loss it assigns falls mostly on low-information tokens like stopwords, punctuation, uncertainty markers, rather than those that determine the answer; within correct rollouts the exploratory tokens incur the highest divergence, so it penalizes the hesitation that reasoning requires. The result is a flatter, less decisive student that is no better at reasoning: as a lone objective, SD optimizes a signal decoupled from task success.

自蒸馏模型偏差推理能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。