跨架构知识蒸馏新方法，显著提升学生模型性能
Multi-Level Decoupled Relational Distillation for Heterogeneous Architectures
- 分层解耦关系对齐，平衡教师模型暗知识与分类置信度
- 多尺度动态融合机制，提升特征层面的知识迁移效果
- 适用于CNN、Transformer等多类架构，泛化性强
异构知识蒸馏能有效将不同架构教师模型的知识传递给学生模型。然而现有方法未能充分挖掘教师输出中的暗知识，限制了性能提升。为此，本文提出多层级解耦关系蒸馏（MLDR-KD）框架，通过在logit和特征层面引入解耦细粒度关系对齐（DFRA），平衡教师模型的暗知识与正确类别置信度之间的权衡。同时设计多尺度动态融合（MSDF）模块，动态融合学生模型不同阶段的多尺度特征投影logits，进一步优化特征级知识迁移。在四种架构（CNN、Transformer、MLP、Mamba）和两个数据集（CIFAR-100、Tiny-ImageNet）上验证，相比最优现有方法，本方法在CIFAR-100上提升最高达4.86%，在Tiny-ImageNet上提升2.78%，展现出良好的鲁棒性与通用性。
原文摘要 · Abstract (English)
Heterogeneous distillation is an effective way to transfer knowledge from cross-architecture teacher models to student models. However, existing heterogeneous distillation methods do not take full advantage of the dark knowledge hidden in the teacher's output, limiting their performance.To this end, we propose a novel framework named Multi-Level Decoupled Relational Knowledge Distillation (MLDR-KD) to unleash the potential of relational distillation in heterogeneous distillation. Concretely, we first introduce Decoupled Finegrained Relation Alignment (DFRA) in both logit and feature levels to balance the trade-off between distilled dark knowledge and the confidence in the correct category of the heterogeneous teacher model. Then, Multi-Scale Dynamic Fusion (MSDF) module is applied to dynamically fuse the projected logits of multiscale features at different stages in student model, further improving performance of our method in feature level. We verify our method on four architectures (CNNs, Transformers, MLPs and Mambas), two datasets (CIFAR-100 and Tiny-ImageNet). Compared with the best available method, our MLDR-KD improves student model performance with gains of up to 4.86% on CIFAR-100 and 2.78% on Tiny-ImageNet datasets respectively, showing robustness and generality in heterogeneous distillation. Code will be released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。