用视觉变压器和同构卷积网络双教师,提升轻量卷积模型视频识别性能。
Revisiting Cross-Architecture Distillation: Adaptive Dual-Teacher Transfer for Lightweight Video Models
- 双教师协同指导:融合视觉变压器与同构卷积教师的预测
- 动态加权融合使监督更精准,最大提升5.95%准确率
- 适合追求高效高精度视频模型的开发者
视觉变换器(ViTs)在视频动作识别中表现优异,但计算成本高;轻量级卷积网络(CNNs)效率高但精度不足。跨架构知识蒸馏(CAKD)通过将ViT知识迁移到CNN来缓解这一问题,但现有方法常因架构差异导致效果不佳,且忽视强同构CNN教师的价值。为此,本文提出双教师知识蒸馏框架,同时利用异构的ViT教师和同构的CNN教师协同指导轻量级CNN学生。引入两个关键组件:(1) 差异感知教师加权,根据教师置信度和与学生预测的差异动态调整权重,实现更有效的监督;(2) 结构差异感知蒸馏策略,通过轻量辅助分支学习ViT与CNN教师之间的残差特征,聚焦可迁移的结构差异,避免模仿ViT全部高维模式。在HMDB51、EPIC-KITCHENS-100和Kinetics-400等基准上广泛实验表明,本方法持续优于现有最优蒸馏方法,在HMDB51上最高获得5.95%的准确率提升。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) have achieved strong performance in video action recognition, but their high computational cost limits their practicality. Lightweight CNNs are more efficient but suffer from accuracy gaps. Cross-Architecture Knowledge Distillation (CAKD) addresses this by transferring knowledge from ViTs to CNNs, yet existing methods often struggle with architectural mismatch and overlook the value of stronger homogeneous CNN teachers. To tackle these challenges, we propose a Dual-Teacher Knowledge Distillation framework that leverages both a heterogeneous ViT teacher and a homogeneous CNN teacher to collaboratively guide a lightweight CNN student. We introduce two key components: (1) Discrepancy-Aware Teacher Weighting, which dynamically fuses the predictions from ViT and CNN teachers by assigning adaptive weights based on teacher confidence and prediction discrepancy with the student, enabling more informative and effective supervision; and (2) a Structure Discrepancy-Aware Distillation strategy, where the student learns the residual features between ViT and CNN teachers via a lightweight auxiliary branch, focusing on transferable architectural differences without mimicking all of ViT's high-dimensional patterns. Extensive experiments on benchmarks including HMDB51, EPIC-KITCHENS-100, and Kinetics-400 demonstrate that our method consistently outperforms state-of-the-art distillation approaches, achieving notable performance improvements with a maximum accuracy gain of 5.95% on HMDB51.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。