让教师温度随样本自适应,提升知识蒸馏效果
Temperature-Adaptive Transformed Teacher Matching

- 基于KL散度最小化,为每个样本动态调整教师温度
- 公式推导出一阶二阶导数,计算高效仅需一次softmax
- 在多个图像分类任务上优于传统自适应方法
温度缩放是知识蒸馏的核心组件,但其作用机制仍不明确。已有的变换教师匹配(TTM)方法仅对教师分布应用温度缩放,并将其目标解释为带有隐式Rényi熵正则化的标准蒸馏。然而,TTM仍依赖固定温度,未说明如何为不同样本调整教师侧温度。本文提出一种样本级逆温度更新机制,通过局部最小化温度缩放后的教师分布与学生预测之间的KL散度来实现。我们推导出逆温度的一阶和二阶闭式导数,其表达式可由变换后教师权重下的中心化教师与学生logits的方差和协方差统计量表示。该方法实现了高效的曲率感知更新,仅需一次softmax计算和常数级别的类别加权求和。在标准图像分类蒸馏基准测试中,所提方法普遍提升了TTM和WTTM性能,同时保持与现有自适应蒸馏基线相当或更优的表现。
原文摘要 · Abstract (English)
Temperature scaling is a core component of knowledge distillation, yet its role and effect are still not fully understood. Transformed Teacher Matching (TTM) clarifies the role of temperature scaling by applying it only to the teacher distribution and interpreting the resulting objective as standard distillation with an implicit Rényi entropy regularization on the student. However, TTM still relies on a fixed temperature and does not specify how the teacher-side temperature should be adapted for individual samples. In this paper, we introduce a sample-wise inverse-temperature update for TTM by locally minimizing the Kullback-Leibler divergence between the temperature-scaled teacher distribution and the student's prediction. We derive closed-form first and second derivatives with respect to the inverse temperature, and show that they can be expressed using variance and covariance statistics of centered teacher and student logits under the transformed teacher weighting. This yields an efficient curvature-aware update that requires one softmax evaluation and a constant number of class-wise weighted sums. Experiments on standard image classification distillation benchmarks show that our temperature adaptation generally improves TTM and WTTM, while remaining competitive with or outperforming prior temperature-adaptive distillation baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。