让学生模型学习教师的类别相对排序，提升细粒度分类能力
Group Relative Knowledge Distillation: Learning from Teacher's Relational Inductive Bias
- 通过相对排名损失，让学生模仿教师的类别间偏好关系
- 在细粒度分类任务上，准确率显著优于传统知识蒸馏方法
- 特别适合需要区分相似类别的场景，如图像细粒度识别
知识蒸馏通常通过最小化师生模型输出分布的差异来迁移知识。然而，现有方法主要关注绝对概率的拟合，忽视了教师模型中蕴含的类别相对预测关系所携带的有益归纳偏置，导致暴露偏差。本文提出一种新型框架——组相对知识蒸馏（GRKD），不再直接拟合绝对分布，而是通过学习类别间的相对排序来蒸馏教师知识。具体而言，引入组相对损失，促使学生模型保留教师输出所提供的成对偏好顺序。在多个分类基准上的大量实验表明，相较于现有方法，GRKD在细粒度分类任务中展现出更优的泛化性能。该方法为挖掘教师知识提供了新视角，聚焦于关系结构而非绝对似然。
原文摘要 · Abstract (English)
Knowledge distillation typically transfers knowledge from a teacher model to a student model by minimizing differences between their output distributions. However, existing distillation approaches largely focus on mimicking absolute probabilities and neglect the valuable relational inductive biases embedded in the teacher's relative predictions, leading to exposure bias. In this paper, we propose Group Relative Knowledge Distillation (GRKD), a novel framework that distills teacher knowledge by learning the relative ranking among classes, rather than directly fitting the absolute distribution. Specifically, we introduce a group relative loss that encourages the student model to preserve the pairwise preference orderings provided by the teacher's outputs. Extensive experiments on classification benchmarks demonstrate that GRKD achieves superior generalization compared to existing methods, especially in tasks requiring fine-grained class differentiation. Our method provides a new perspective on exploiting teacher knowledge, focusing on relational structure rather than absolute likelihood.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。