用排序理论重构知识蒸馏,让模型更精准学习教师的分类信心排序。
PLD: A Choice-Theoretic List-Wise Knowledge Distillation

- 将教师输出视为类别价值,构建基于排名的加权损失函数
- 在多个数据集上超越传统蒸馏方法,提升稳定性和准确性
- 适合需要高质量知识迁移的模型压缩场景
知识蒸馏通过训练小型学生网络模仿大型教师网络的预测行为实现模型压缩。传统基于logit的蒸馏方法通常在交叉熵基础上添加一个正则项,如KL散度或相关性损失,但需手动调节权重。本文从选择理论出发,将教师logits建模为类别“价值”得分,提出Plackett-Luce蒸馏(PLD),一种加权列表级排序损失。PLD直接优化教师最优排序:真标签排第一,其余按教师置信度降序排列。该方法构建了一个凸且平移不变的代理损失,可统一加权交叉熵。在CIFAR-100、ImageNet-1K和MS-COCO上,无论架构或蒸馏目标(发散、相关性、特征匹配),PLD均在同构与异构师生对中持续取得性能提升。
原文摘要 · Abstract (English)
Knowledge distillation is a model compression technique in which a compact "student" network is trained to replicate the predictive behavior of a larger "teacher" network. In logit-based knowledge distillation, it has become the de facto approach to augment cross-entropy with a distillation term. Typically, this term is either a KL divergence that matches marginal probabilities or a correlation-based loss that captures intra- and inter-class relationships. In every case, it acts as an additional term to cross-entropy. This term has its own weight, which must be carefully tuned. In this paper, we adopt a choice-theoretic perspective and recast knowledge distillation under the Plackett-Luce model by interpreting teacher logits as "worth" scores. We introduce "Plackett-Luce Distillation (PLD)", a weighted list-wise ranking loss. In PLD, the teacher model transfers knowledge of its full ranking of classes, weighting each ranked choice by its own confidence. PLD directly optimizes a single "teacher-optimal" ranking. The true label is placed first, followed by the remaining classes in descending teacher confidence. This process yields a convex and translation-invariant surrogate that subsumes weighted cross-entropy. Empirically, across CIFAR-100, ImageNet-1K, and MS-COCO, PLD achieves consistent gains across diverse architectures and distillation objectives, including divergence-based, correlation-based, and feature-based methods, in both homogeneous and heterogeneous teacher-student pairs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。