通过交换逻辑值提升教师学生对齐，显著改善模型压缩效果
Swapped Logit Distillation via Bi-level Teacher Alignment
- 用交换教师与学生输出的逻辑值构建双教师机制
- 在图像分类任务中优于现有最先进方法，性能稳定提升
- 适合需要高效模型压缩的落地场景，如移动端部署
知识蒸馏通过将大模型（教师）的知识迁移到小模型（学生）来压缩网络容量。当前主流方法直接使用教师原始输出分布进行迁移，可能引发错误预测。本文提出基于逻辑值交换的蒸馏方法——交换逻辑蒸馏（SLD），基于两个假设：(1) 当预测标签置信度非最大时会出现错误预测；(2) 概率的“自然”上限尚不明确，无法确定最优目标添加值。为此，我们设计了交换逻辑处理方案，使该方法可有效扩展至教师和学生的输出，形成两个教师。进一步引入损失调度策略以增强双教师对齐。大量图像分类实验表明，SLD在多个基准上持续优于现有最先进方法。
原文摘要 · Abstract (English)
Knowledge distillation (KD) compresses the network capacity by transferring knowledge from a large (teacher) network to a smaller one (student). It has been mainstream that the teacher directly transfers knowledge to the student with its original distribution, which can possibly lead to incorrect predictions. In this article, we propose a logit-based distillation via swapped logit processing, namely Swapped Logit Distillation (SLD). SLD is proposed under two assumptions: (1) the wrong prediction occurs when the prediction label confidence is not the maximum; (2) the "natural" limit of probability remains uncertain as the best value addition to the target cannot be determined. To address these issues, we propose a swapped logit processing scheme. Through this approach, we find that the swap method can be effectively extended to teacher and student outputs, transforming into two teachers. We further introduce loss scheduling to boost the performance of two teachers' alignment. Extensive experiments on image classification tasks demonstrate that SLD consistently performs best among previous state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。