用语义关系提升模型压缩效果，让小模型学得更准。
Delving Deep into Semantic Relation Distillation
- 用超像素提取语义，通过样本间关系传递知识
- 在ImageNet等数据集上显著优于现有方法
- 特别适合视觉Transformer的轻量化设计
知识蒸馏已成为深度学习中的核心技术，实现复杂模型向轻量级模型的知识迁移。传统方法仅关注实例级知识传递，难以捕捉数据内部的细微语义关系。为此，本文提出一种新方法——基于语义关系的知识蒸馏（SeRKD），从样本间的语义关系视角重构知识蒸馏过程。通过引入语义单元（即超像素），SeRKD实现了更全面、上下文感知的知识迁移，巧妙融合了基于超像素的语义提取与关系型知识蒸馏，显著提升模型压缩与蒸馏效果。该方法在视觉变压器（ViTs）中尤为自然适用，因视觉标记本身就是核心表示单元。在多个基准数据集上的实验表明，SeRKD在模型性能和泛化能力上均优于现有方法。
原文摘要 · Abstract (English)
Knowledge distillation has become a cornerstone technique in deep learning, facilitating the transfer of knowledge from complex models to lightweight counterparts. Traditional distillation approaches focus on transferring knowledge at the instance level, but fail to capture nuanced semantic relationships within the data. In response, this paper introduces a novel methodology, Semantics-based Relation Knowledge Distillation (SeRKD), which reimagines knowledge distillation through a semantics-relation lens among each sample. By leveraging semantic components, \ie, superpixels, SeRKD enables a more comprehensive and context-aware transfer of knowledge, which skillfully integrates superpixel-based semantic extraction with relation-based knowledge distillation for a sophisticated model compression and distillation. Particularly, the proposed method is naturally relevant in the domain of Vision Transformers (ViTs), where visual tokens serve as fundamental units of representation. Experimental evaluations on benchmark datasets demonstrate the superiority of SeRKD over existing methods, underscoring its efficacy in enhancing model performance and generalization capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。