通过排序机制改进教师模型输出，让学生模型学得更准
Parameter-Free Logit Distillation via Sorting Mechanism
- 用排序方法修正教师模型的错误预测，重排输出分布
- 在CIFAR-100和ImageNet上显著提升学生模型准确率
- 无需额外参数，可直接插入现有知识蒸馏流程
知识蒸馏（KD）旨在通过软标签将教师模型（较大）的知识传递给学生模型（较小），以实现高效神经网络。通常模型性能由准确率衡量，依赖真实标签。然而，现有KD方法常直接使用教师模型原始输出分布，忽略了其错误预测的潜在影响。这与交叉熵损失引导硬标签学习的初衷相悖，可能导致某些样本上的知识蒸馏效果不佳。为此，我们提出一种基于排序机制的新式对数处理方案。该方法具有双重目标：(1) 根据真实标签修正教师模型的错误预测；(2) 一次性按优先级顺序重新排列输出分布。作为一种无需参数、即插即用的预处理方法，该排序策略可有效应用于现有的基于对数的KD方法。在CIFAR-100和ImageNet数据集上的大量实验验证了该方法的有效性。
原文摘要 · Abstract (English)
Knowledge distillation (KD) aims to distill the knowledge from the teacher (larger) to the student (smaller) model via soft-label for the efficient neural network. In general, the performance of a model is determined by accuracy, which is measured with labels. However, existing KD approaches usually use the teacher with its original distribution, neglecting the potential of incorrect prediction. This may contradict the motivation of hard-label learning through cross-entropy loss, which may lead to sub-optimal knowledge distillation on certain samples. To address this issue, we propose a novel logit processing scheme via a sorting mechanism. Specifically, our method has a two-fold goal: (1) fixing the incorrect prediction of the teacher based on the labels and (2) reordering the distribution in a natural way according to priority rank at once. As an easy-to-use, plug-and-play pre-processing, our sort method can be effectively applied to existing logit-based KD methods. Extensive experiments on the CIFAR-100 and ImageNet datasets demonstrate the effectiveness of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。