通过校准教师模型输出，解决知识蒸馏中的误导问题。
LoCa: Logit Calibration for Knowledge Distillation
- 基于真实标签修正教师模型的logits，避免学生模型被误导
- 不增加参数即可提升图像分类与文本生成任务性能
- 适合追求轻量高效蒸馏方案的研究者
知识蒸馏（KD）旨在通过模仿教师模型来训练更优的学生模型，在模型压缩中具有重要作用。通常方法是对齐输出logits，但我们发现一个常见问题：当基于教师logits的预测与真实标签不符时，学生模型会受到误导。此外，logits中还蕴含类间可区分性等有用‘暗知识’，对蒸馏至关重要。本文提出一种简单而有效的Logit Calibration（LoCa）方法，基于真实标签对教师模型的logits进行校准。核心思想是同时纠正错误预测（解决误导问题）并保留有用的暗知识。该方法无需额外参数。在图像分类和文本生成任务上的实验证明，LoCa能显著提升基线模型性能。
原文摘要 · Abstract (English)
Knowledge Distillation (KD), aiming to train a better student model by mimicking the teacher model, plays an important role in model compression. One typical way is to align the output logits. However, we find a common issue named mis-instruction, that the student would be misled when the predictions based on teacher logits do not follow the labels. Meanwhile, there is other useful dark knowledge in the logits such as the class discriminability, which is vital for distillation. In this paper, we propose a simple yet effective Logit Calibration (LoCa) method, which calibrates the logits from the teacher model based on the ground-truth labels. The key insight is to correct the prediction (to address the mis-instruction issue) and maintain useful dark knowledge simultaneously. Our proposed LoCa does not require any additional parameters. Empirical results on image classification and text generation tasks demonstrate that LoCa can effectively improve the performance of baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。