根据教师模型对数输出相关性动态调整温度,提升知识蒸馏效果
Adaptive Temperature Based on Logits Correlation in Knowledge Distillation
- 基于教师模型最大对数输出计算动态温度,减少计算开销
- 在标准数据集上验证,动态温度使学生模型性能优于静态设置
- 适用于各类师生模型组合,尤其适合追求高效蒸馏的场景
知识蒸馏通过将教师模型的输出用于训练更小的学生模型,实现性能压缩。传统方法使用固定温度参数调节软标签分布,但其如何促进信息传递尚不明确。本文提出一种新方法,仅依据教师模型生成的最大对数输出计算动态温度,显著降低计算成本。实验表明,在标准基准数据集上,该方法在多种师生模型组合中均取得更优性能。动态温度估计值高于常用静态值,且蒸馏过程逼近两模型对数间的相关性,强化了对数相关性在知识传递中的关键作用。
原文摘要 · Abstract (English)
Knowledge distillation is a technique to imitate a performance that a deep learning model has, but reduce the size on another model. It applies the outputs of a model to train another model having comparable accuracy. These two distinct models are similar to the way information is delivered in human society, with one acting as the "teacher" and the other as the "student". Softmax plays a role in comparing logits generated by models with each other by converting probability distributions. It delivers the logits of a teacher to a student with compression through a parameter named temperature. Tuning this variable reinforces the distillation performance. Although only this parameter helps with the interaction of logits, it is not clear how temperatures promote information transfer. In this paper, we propose a novel approach to calculate the temperature. Our method only refers to the maximum logit generated by a teacher model, which reduces computational time against state-of-the-art methods. Our method shows a promising result in different student and teacher models on a standard benchmark dataset. Algorithms using temperature can obtain the improvement by plugging in this dynamic approach. Furthermore, the approximation of the distillation process converges to a correlation of logits by both models. This reinforces the previous argument that the distillation conveys the relevance of logits. We report that this approximating algorithm yields a higher temperature compared to the commonly used static values in testing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。