让偏倚的教师模型教出均衡知识，提升长尾分布下的分类效果。
Distilling Balanced Knowledge from a Biased Teacher
- 拆解教师模型偏差来源，分组设计跨组与组内损失
- 在多个长尾数据集上，尾部类别准确率显著提升
- 适合需要公平分类性能的现实场景应用
传统知识蒸馏在长尾分布上表现不佳，因教师模型对头部类别过度偏好，难以有效指导尾部类别。本文提出长尾知识蒸馏（LTKD），将传统目标分解为跨组损失（捕捉头、中、尾类间预测分布差异）和组内损失（捕捉组内分布差异）。该分解揭示了教师模型偏倚的具体来源。为缓解继承性偏倚，LTKD引入重平衡的跨组损失以校准教师组级预测，并采用重加权的组内损失确保各组贡献均等。在CIFAR-100-LT、TinyImageNet-LT和ImageNet-LT上的大量实验表明，LTKD在整体与尾部类别准确率上均显著优于现有方法，证明其能从有偏教师中蒸馏出均衡知识，适用于真实世界应用。
原文摘要 · Abstract (English)
Conventional knowledge distillation, designed for model compression, fails on long-tailed distributions because the teacher model tends to be biased toward head classes and provides limited supervision for tail classes. We propose Long-Tailed Knowledge Distillation (LTKD), a novel framework that reformulates the conventional objective into two components: a cross-group loss, capturing mismatches in prediction distributions across class groups (head, medium, and tail), and a within-group loss, capturing discrepancies within each group's distribution. This decomposition reveals the specific sources of the teacher's bias. To mitigate the inherited bias, LTKD introduces (1) a rebalanced cross-group loss that calibrates the teacher's group-level predictions and (2) a reweighted within-group loss that ensures equal contribution from all groups. Extensive experiments on CIFAR-100-LT, TinyImageNet-LT, and ImageNet-LT demonstrate that LTKD significantly outperforms existing methods in both overall and tail-class accuracy, thereby showing its ability to distill balanced knowledge from a biased teacher for real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。