知识蒸馏要评估丢失的能力,不能只看任务得分。
Knowledge Distillation Must Account for What It Loses
- 把蒸馏看作有损压缩,关注学生模型失去的教师能力。
- 发现当前评估常忽略能力损失,导致结果不可靠。
- 提出保留目标与损失声明,让蒸馏过程可问责。
本文主张知识蒸馏必须正视其损失:学生模型不应仅以任务得分评判,还需检验是否保留了使教师表现可靠的底层能力。这很重要,因为蒸馏正被广泛用于将大模型转化为可部署的小模型,但主流指标常掩盖关键能力的丧失。本文指出,现有评估隐含假设——保留任务得分即保留教师能力,这是错误的。将蒸馏视为有损投影,揭示学生可能匹配部分可观测行为,却未保留使其可靠的能力基础。我们整合已有证据,构建了一个离指标蒸馏损失的分类体系,表明这些损失是真实、反复出现且可测量的,但常被研究忽视。为推动实践,我们提出场景特定的保留目标和‘蒸馏损失声明’,明确报告保留了什么、丢失了什么,以及剩余损失为何可接受。目标不是无损蒸馏,而是可问责的蒸馏。
原文摘要 · Abstract (English)
This position paper argues that knowledge distillation must account for what it loses: student models should be judged not only by retained task scores, but by whether they preserve the teacher capabilities that make those scores reliable. This matters because distillation is increasingly used to turn large teacher models into deployable students, yet headline metrics can obscure losses in the capabilities that make teacher behavior reliable. Conceptually, we show that current evaluation often assumes retained task scores imply retained teacher capabilities. Reframing distillation as a lossy projection exposes this flaw: students may match selected teacher observables without preserving the capabilities that make them reliable. We then synthesize existing evidence into a taxonomy of off-metric distillation losses, showing that such losses are concrete, recurring, and measurable, yet often unaccounted for when studies report what students retain rather than what they lose. To make the position actionable, we propose scenario-specific preservation targets and a Distillation Loss Statement that reports what was preserved, what was lost, and why the remaining losses are acceptable. The goal is not lossless distillation, but accountable distillation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。