用排序损失提升知识蒸馏，让模型学得更均衡。
Enhancing Logits Distillation with Plug\&Play Kendall's $τ$ Ranking Loss
- 引入肯德尔τ排序损失，重构类别间关系信息
- 梯度更均衡，低概率通道更新更强，性能提升显著
- 即插即用，适配各类蒸馏框架与模型结构
知识蒸馏通常通过最小化教师和学生输出对数的KL散度来实现。然而，优化KL散度对学生而言颇具挑战，常导致次优解。我们发现KL梯度随教师对数幅度缩放，导致低概率通道更新减弱，削弱了类别间信息传递，限制了学生性能提升。为此，我们提出一种即插即用的辅助排序损失，基于肯德尔τ系数，可无缝集成于任意基于对数的蒸馏框架中。该损失提供类别间相对关系信息，同时重平衡低概率通道的梯度更新。实验表明，所提损失对通道缩放基本不变，且其优化目标与KL散度一致，是天然补充而非替代。在CIFAR-100、ImageNet和COCO数据集上，多种CNN与ViT架构组合下，本方法持续提升多个蒸馏基线性能。代码已开源。
原文摘要 · Abstract (English)
Knowledge distillation typically minimizes the Kullback-Leibler (KL) divergence between teacher and student logits. However, optimizing the KL divergence can be challenging for the student and often leads to sub-optimal solutions. We further show that gradients induced by KL divergence scale with the magnitude of the teacher logits, thereby diminishing updates on low-probability channels. This imbalance weakens the transfer of inter-class information and in turn limits the performance improvements achievable by the student. To mitigate this issue, we propose a plug-and-play auxiliary ranking loss based on Kendall's $τ$ coefficient that can be seamlessly integrated into any logit-based distillation framework. It supplies inter-class relational information while rebalancing gradients toward low-probability channels. We demonstrate that the proposed ranking loss is largely invariant to channel scaling and optimizes an objective aligned with that of KL divergence, making it a natural complement rather than a replacement. Extensive experiments on CIFAR-100, ImageNet, and COCO datasets, as well as various CNN and ViT teacher-student architecture combinations, demonstrate that our plug-and-play ranking loss consistently boosts the performance of multiple distillation baselines. Code is available at https://github.com/OvernighTea/RankingLoss-KD
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。