让弱教师辅助强学生，训练速度最高提升4.8倍。
Weak-to-Strong Knowledge Distillation Accelerates Visual Learning
- 用弱教师在早期训练中指导强学生，达标后停止
- 图像分类任务中最快提前4.8倍完成训练
- 适用于检测与生成任务，适合追求高效训练的研究者
大规模视觉学习正面临训练成本上升的瓶颈。现有知识蒸馏方法通常将知识从强教师传给弱学生，用于压缩或提升精度。本文反其道而行之，研究如何利用蒸馏加速强学生的训练过程。提出一种通用、即插即用的方案：冻结一个弱教师，在训练早期阶段进行蒸馏，一旦学生性能达到并超过教师水平便停止蒸馏。在ImageNet和CIFAR分类任务中，该策略显著提前达到目标准确率，最快可节省4.8倍训练轮次。该方法在其他任务上也表现良好：在COCO目标检测任务中实现1.7倍轮次加速，在CIFAR-10扩散生成任务中实现2.5倍步数提前达成目标FID。这些结果验证了本方法是视觉学习中的通用加速机制。
原文摘要 · Abstract (English)
Large-scale visual learning is increasingly limited by training cost. Existing knowledge distillation methods transfer from a stronger teacher to a weaker student for compression or final-accuracy improvement. We instead investigate distillation to accelerate the training of strong students. We propose a generalizable plug-and-play recipe that freezes a weaker teacher, applies distillation only in early training, and turns it off once the student reaches and surpasses teacher-level performance. For ImageNet and CIFAR classification, this strategy reaches target thresholds much earlier, with up to 4.8 times speedup measured by epochs. We confirm that the method generalizes to other tasks and report 1.7 times epoch speedup for object detection on the COCO dataset, and 2.5 times earlier target-FID crossing for diffusion generation on the CIFAR-10 dataset, measured in steps. These findings validate our method as a universal speedup mechanism for visual learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。