用频域特征统一异构模型知识蒸馏,提升跨架构迁移效果。
UHKD: A Unified Framework for Heterogeneous Knowledge Distillation via Frequency-Domain Representations
- 将教师模型中间特征转为频域表示,缓解异构架构间的语义差异。
- 在CIFAR-100和ImageNet-1K上分别提升5.59%和0.83%准确率。
- 适合需要跨架构压缩的轻量级模型部署场景。
知识蒸馏(KD)是一种有效的模型压缩技术,可将高性能教师模型的知识迁移到轻量级学生模型中,降低计算与存储开销,同时保持较高精度。然而,现有大多数KD方法针对同构模型设计,在涉及中间特征的异构设置下表现不佳。不同架构间的语义差异阻碍了教师模型中间表示的有效利用,而以往异构KD研究多集中于输出层(logits空间),未能充分挖掘中间层丰富的语义信息。为此,本文提出统一异构知识蒸馏(UHKD)框架,通过频域表示实现跨架构中间特征迁移。频域表示能捕捉全局语义并缓解异构师生对之间的表征差异。具体而言,特征转换模块(FTM)生成教师特征的紧凑频域表示,可学习的特征对齐模块(FAM)通过多层次匹配将学生特征投影并对齐。训练采用联合目标:中间特征上的均方误差与输出层上的KL散度。在CIFAR-100和ImageNet-1K上的大量实验表明,该方法优于最新异构蒸馏方法,在两个数据集上分别取得最大5.59%和0.83%的准确率提升。代码将尽快公开。
原文摘要 · Abstract (English)
Knowledge distillation (KD) is an effective model compression technique that transfers knowledge from a high-performance teacher to a lightweight student, reducing computational and storage costs while maintaining competitive accuracy. However, most existing KD methods are tailored for homogeneous models and perform poorly in heterogeneous settings, particularly when intermediate features are involved. Semantic discrepancies across architectures hinder effective use of intermediate representations from the teacher model, while prior heterogeneous KD studies mainly focus on the logits space, underutilizing rich semantic information in intermediate layers. To address this, Unified Heterogeneous Knowledge Distillation (UHKD) is proposed, a framework that leverages intermediate features in the frequency domain for cross-architecture transfer. Frequency-domain representations are leveraged to capture global semantic knowledge and mitigate representational discrepancies between heterogeneous teacher-student pairs. Specifically, a Feature Transformation Module (FTM) generates compact frequency-domain representations of teacher features, while a learnable Feature Alignment Module (FAM) projects student features and aligns them via multi-level matching. Training is guided by a joint objective combining mean squared error on intermediate features with Kullback-Leibler divergence on logits. Extensive experiments on CIFAR-100 and ImageNet-1K demonstrate the effectiveness of the proposed approach, achieving maximum gains of 5.59% and 0.83% over the latest heterogeneous distillation method on the two datasets, respectively. Code will be released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。