动态调整温度提升知识蒸馏效果
Dynamic Temperature Scheduler for Knowledge Distillation
- 根据师生模型差异自动调节温度，不再固定
- 在图像和文本任务上均显著优于固定温度
- 适合追求高精度蒸馏的开发者使用
知识蒸馏（KD）通过预训练的大模型（教师）指导小模型（学生）训练，温度是控制输出概率软度的关键超参数。传统方法采用固定温度，效果不佳；且师生模型结构差异常导致对数几率幅度不匹配。我们发现学生模型早期需要更柔和的概率分布，后期则需更尖锐的分布。为此提出动态温度调度器（DTS），根据师生模型之间的交叉熵损失差距动态调整温度。据我们所知，这是首个基于师生分布差异自适应调整温度的方法。DTS可无缝集成至现有KD框架中。我们在视觉（CIFAR-100、Tiny-ImageNet）和自然语言处理任务（GLUE、Dolly、SelfIns、UnNI、S-NI）上验证了该方法，结果一致优于静态温度基线。代码已开源：https://github.com/Sibgat-Ul/DTS。
原文摘要 · Abstract (English)
Knowledge Distillation (KD) trains a smaller student model using a large, pre-trained teacher model, with temperature as a key hyperparameter controlling the softness of output probabilities. Traditional methods use a fixed temperature throughout training, which is suboptimal. Moreover, architectural differences between teacher and student often result in mismatched logit magnitudes. We demonstrate that students benefit from softer probabilities early in training but require sharper probabilities in later stages. We introduce Dynamic Temperature Scheduler (DTS), which adjusts temperature dynamically based on the cross-entropy loss gap between teacher and student. To our knowledge, this is the first temperature scheduling method that adapts based on the divergence between teacher and student distributions. Our method integrates seamlessly with existing KD frameworks. We validate DTS across multiple KD strategies on vision (CIFAR-100, Tiny-ImageNet) and NLP tasks (GLUE, Dolly, SelfIns, UnNI, S-NI), consistently outperforming static-temperature baselines. Code is available at https://github.com/Sibgat-Ul/DTS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。