让不同架构的模型互相教，提升知识蒸馏效果
Perspective-Aware Teaching: Adapting Knowledge for Heterogeneous Distillation
- 用提示调优让教师特征适应学生学习过程
- 通过区域感知注意力解决视角不匹配问题
- 适用于跨架构蒸馏，尤其适合异构模型
知识蒸馏（KD）通过将预训练大型教师模型的知识迁移到轻量级学生模型，降低推理成本同时保持性能。传统方法假设教师与学生模型同构，但随着技术发展，从卷积神经网络（CNNs）到视觉变换器（ViTs）和多层感知机（MLPs）等多样化架构涌现，开发兼容任意架构的通用蒸馏框架成为关键课题。本文提出视角感知教学（PAT）框架，实现跨异构架构的特征蒸馏。该框架包含两个核心组件：首先设计结合学生反馈的提示调优模块，使教师特征能动态适配学生学习进程；其次提出区域感知注意力机制，缓解异构架构间的视图不匹配问题。在CIFAR、ImageNet和COCO上的大量实验表明，所提方法显著优于现有方法。代码已开源。
原文摘要 · Abstract (English)
Knowledge distillation (KD) involves transferring knowledge from a pre-trained heavy teacher model to a lighter student model, thereby reducing the inference cost while maintaining comparable effectiveness. Prior KD techniques typically assume homogeneity between the teacher and student models. However, as technology advances, a wide variety of architectures have emerged, ranging from initial Convolutional Neural Networks (CNNs) to Vision Transformers (ViTs), and Multi-Level Perceptrons (MLPs). Consequently, developing a universal KD framework compatible with any architecture has become an important research topic. In this paper, we introduce a perspective-aware teaching (PAT) KD framework to enable feature distillation across diverse architectures. Our framework comprises two key components. First, we design prompt tuning blocks that incorporate student feedback, allowing teacher features to adapt to the student model's learning process. Second, we propose region-aware attention to mitigate the view mismatch problem between heterogeneous architectures. By leveraging these two modules, effective distillation of intermediate features can be achieved across heterogeneous architectures. Extensive experiments on CIFAR, ImageNet, and COCO demonstrate the superiority of the proposed method. Our code is available at https://github.com/jimmylin0979/PAT.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。