通过强化重要输出信息，提升模型压缩效果。
TopKD: Top-scaled Knowledge Distillation
- 自适应放大教师模型最显著的预测结果，聚焦关键知识。
- 在多个数据集上超越现有方法，最高提升2.1%准确率。
- 无需修改结构，适合各类模型和蒸馏框架使用。
近期知识蒸馏研究多关注特征级知识迁移，常忽略教师模型输出概率分布中的关键信息。本文重新审视基于logit的蒸馏，揭示了一个未被充分探索但至关重要的元素：Top-K知识。受此启发，提出一种简单、高效且架构无关的框架——顶尺度知识蒸馏（TopKD），显著提升基于logit的蒸馏性能。TopKD包含两个核心组件：(1) 顶-K缩放模块（TSM），自适应增强最具信息量的logits；(2) 顶-K解耦损失（TDL），提供精准有效的监督信号。值得注意的是，TopKD可无缝集成至现有蒸馏方法中，无需额外模块或架构改动。在CIFAR-100、ImageNet、STL-10和Tiny-ImageNet上的大量实验表明，TopKD始终优于当前最优蒸馏方法。此外，该方法在蒸馏Vision Transformers时也表现优异，凸显其在多种网络架构下的普适性。这些发现彰显了logits在推动知识蒸馏发展中的巨大潜力。
原文摘要 · Abstract (English)
Recent advances in knowledge distillation (KD) predominantly emphasize feature-level knowledge transfer, frequently overlooking critical information embedded within the teacher's logit distributions. In this paper, we revisit logit-based distillation and reveal an underexplored yet critical element: Top-K knowledge. Motivated by this insight, we propose Top-scaled Knowledge Distillation (TopKD), a simple, efficient, and architecture-agnostic framework that significantly enhances logit-based distillation. TopKD consists of two main components: (1) a Top-K Scaling Module (TSM), which adaptively amplifies the most informative logits, and (2) a Top-K Decoupled Loss (TDL), which offers targeted and effective supervision. Notably, TopKD integrates seamlessly into existing KD methods without introducing extra modules or requiring architectural changes. Extensive experiments on CIFAR-100, ImageNet, STL-10, and Tiny-ImageNet demonstrate that TopKD consistently surpasses state-of-the-art distillation methods. Moreover, our method demonstrates substantial effectiveness when distilling Vision Transformers, underscoring its versatility across diverse network architectures. These findings highlight the significant potential of logits to advance knowledge distillation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。