通过分层跳过机制，让小模型更快学懂大模型的视觉特征。
LEAP: Layer-skipping Efficiency via Adaptive Progression for Vision Transformer Distillation
- 用教师模型中间层特征做渐进式训练目标，逐步提升难度。
- 小模型在ImageNet-100上准确率提升12.24%，达90.1%。
- 训练效率提升，节省25%算力和21%时间，适合边缘部署。
基于视觉变换器（ViT）的视觉基础模型（如DINOv2）在目标识别、语义分割等下游任务中至关重要。由于骨干网络计算量巨大，常需通过知识蒸馏压缩至小型架构以实现边缘部署。基于特征的知识蒸馏常因教师-学生能力差距而受限：学生因容量不足难以模仿教师复杂特征图。为此，本文提出LEAP——一种面向ViT特征蒸馏的自适应进度训练课程。通过将教师中间特征图作为一系列逐步增强难度的目标，该课程使学生先建立基础表征，再学习高层抽象。实验表明，该范式显著加速收敛，适用于不同规模的学生模型与数据集。使用该课程后，LEAP蒸馏的ViT-S在ImageNet-100上达到90.1%准确率，较基线提升12.24%；在ImageNet-1K上，对Oxford与Paris数据集的实例检索任务分别提升3.84%与7.75%。此外，通过在训练初期阶段对教师推理实施早停，实现25.1%的训练浮点运算量（FLOPs）节省与21%的训练时间节省。代码已开源。
原文摘要 · Abstract (English)
Vision Foundation Models (VFMs) with Vision Transformer (ViT) backbones, such as DINOv2, have become essential for downstream tasks like object recognition and semantic segmentation. The immense computational requirements of backbones often necessitate distillation into smaller architectures for edge deployment. Feature-based knowledge distillation (KD) often suffers from the teacher-student gap; the student struggles to imitate teacher's complex feature map due to its limited capacity. To mitigate this bottleneck, we propose LEAP: Layer-skipping Efficiency via Adaptive Progression, a training curriculum for ViT feature-based knowledge distillation. By utilizing the teacher's intermediate feature maps as a sequence of progressively more difficult targets, our curriculum allows the student to build a foundational representation before tackling higher-level abstractions. Our results demonstrate that this paradigm significantly accelerates convergence through adaptive difficulty selection across various student model sizes and dataset scales. With our curriculum, the LEAP-distilled ViT-S achieves 90.1% accuracy on ImageNet-100, a +12.24% improvement compared with baseline. On ImageNet-1K, LEAP achieves +3.84% and +7.75% improvement for the instance retrieval task on the Oxford and Paris datasets, respectively. Furthermore, the curriculum enables 25.1% savings in training FLOPs and 21% savings in training time on ImageNet-100 by implementing early-stopping for teacher inference during the initial stages of training. Code is available at https://github.com/KevinZ0217/LEAP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。