老师出错时,学生模仿越像反而任务表现越差,需警惕表面成功。
Knowledge Distillation under Teacher Misspecification: An Order-Parameter Analysis of the Gap between Teacher Mimicry and Task Performance

- 用三体模型分析教师错配对知识蒸馏的影响
- 教师模仿误差不变,真实任务误差随错配程度线性上升
- 提出误差差值Δ作为诊断教师错配的关键指标
知识蒸馏通常通过教师-学生输出差异来监控训练进展,但真正关心的是学生在真实任务上的误差。本文构建一个最小三体模型:真实教师(生成模型)、教师和学生均为软委员会机,其中真实教师包含教师无法表征的共享潜变量,错配强度由标量$\ ext{dmiss}$控制。在在线蒸馏的序参数描述下,利用误差函数激活下的闭式(弧正弦型)表达式,证明学习动态和蒸馏误差$\ ext{Ets}$严格不随$\ ext{dmiss}$变化,而真实误差$\ ext{Etzs}$与差距$\ ext{Δ} = \text{Etzs} - \text{Ets}$则随$\ ext{dmiss}$严格递增,且增长率被真实教师复杂度$M_0$线性放大。数值相图验证了预测:$\ ext{Ets}$等值线不变,$\ ext{Etzs}$景观整体抬升,教师错配区(模仿成功但任务失败)随$\ ext{dmiss}$扩大。结果警示不应仅依赖模仿误差评估蒸馏效果,并将$\ ext{Δ}$定位为区分教师错配与容量不足失败的最小诊断工具。
原文摘要 · Abstract (English)
Knowledge distillation trains a small student model to reproduce the outputs of a large teacher model, and its progress is typically monitored through the teacher--student discrepancy. The quantity of ultimate interest, however, is the student's error with respect to the true task. We study the relation between these two objectives in a minimal three-party model, a true teacher (generative model), a teacher, and a student, all soft committee machines, in which the true teacher contains a shared latent factor that the teacher cannot represent, with mismatch strength controlled by a single scalar $\dmiss$. Within an order-parameter description of online distillation, and exploiting closed-form (arcsine-type) expressions for all errors under error-function activations, we prove that the learning dynamics and the distillation error $\Ets$ are exactly invariant to $\dmiss$, whereas the true error $\Etzs$ and the gap $Δ=\Etzs-\Ets$ are strictly increasing in $\dmiss$, with a rate that is amplified linearly by the complexity $M_0$ of the true teacher. Numerical phase diagrams over the plane spanned by true-teacher complexity and student capacity confirm the predicted deformation: the contours of $\Ets$ do not move while the landscape of $\Etzs$ rises systematically, and a teacher-miss regime, where mimicry succeeds but the task fails, expands with $\dmiss$. The results give a quantitative warning against evaluating distillation solely through teacher-mimicry metrics and identify the gap $Δ$ as a minimal diagnostic for distinguishing teacher-miss from capacity-limited failure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。