arXiv:2607.11557cs.CV2026-07

用学生模型动态生成教师视角扰动,提升知识蒸馏效率与效果

Single-Teacher View Augmentation: Enhancing Knowledge Distillation with Student-Guided Perturbations

论文配图:Single-Teacher View Augmentation: Enhancing Knowledge Distillation with Student-Guided Perturbations
图 1 · 摘自论文原文
  • 以学生模型的演化特征为条件生成扰动,实现动态视角增强
  • 单阶段训练下性能媲美两阶段方法,参数量减少90%以上
  • 无需预训练,适合资源受限场景下的高效模型压缩

知识蒸馏通常依赖单一教师模型的固定视角,限制了监督信号的多样性。虽然多教师蒸馏能提升多样性,但带来高昂的计算与存储开销。现有单教师虚拟视图生成方法存在权衡:随机扰动效率高但缺乏可控多样性,结构化增强需多阶段训练且参数线性增长。本文发现该困境源于使用静态教师特征生成视图。为此提出Shift-Augmented KD(SAKD),利用学生模型演化特征作为扰动生成的动态条件,实现单阶段训练。通过无参循环移位生成自适应、多样化的视图。在CIFAR-100和ImageNet上的实验表明,SAKD持续优于随机扰动方法,性能接近两阶段方案,同时参数量显著降低,且无需预训练。

原文摘要 · Abstract (English)

Knowledge distillation (KD) typically relies on the fixed perspective of a single teacher, limiting the diversity of supervisory signals. While multi-teacher distillation addresses this by aggregating knowledge from multiple models, it incurs prohibitive computational and storage costs. To balance efficiency and diversity, recent research has focused on generating virtual views from a single teacher. However, existing methods face a trade-off: random perturbation approaches offer efficiency but lack controlled diversity, while structured augmentation methods require multi-stage training and incur linear parameter growth. We observe that this trade-off stems from a common design choice: using the teacher's strong but static features to generate views. Instead, we propose Shift-Augmented Knowledge Distillation (SAKD), a simple yet effective framework that leverages the student's evolving features as a dynamic condition for perturbation generation. This shift in perspective enables single-stage training while producing adaptive, diverse views through a parameter-free cyclic shift. Extensive experiments on CIFAR-100 and ImageNet demonstrate that SAKD consistently outperforms random perturbation methods and achieves accuracy on par with two-stage approaches, while using significantly fewer parameters and eliminating pre-training requirements.

知识蒸馏动态增强模型压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。