通过模仿生物学习的间隔效应，提升知识蒸馏的泛化能力。
Right Time to Learn:Promoting Generalization via Bio-inspired Spacing Effect in Knowledge Distillation
- 学生模型从提前间隔训练的教师模型中蒸馏知识，模拟生物学习规律。
- 在Tiny-ImageNet上相比在线蒸馏和自蒸馏，性能提升最高达3.34%。
- 适合希望提升模型泛化性且不改变原有框架的研究者使用。
知识蒸馏（KD）是训练深度神经网络的强大策略。尽管最初用于从大型教师模型中训练更紧凑的学生模型，但近年来许多研究将其用于提升模型自身的泛化能力，如在线KD和自蒸馏。本文提出一种简单且兼容的策略——间隔蒸馏（Spaced KD），其中学生模型从一个提前间隔训练的教师模型中蒸馏知识。该策略受生物学习中的间隔效应启发，认为学习试验之间的适当间隔可显著提升学习效果。通过理论与实证分析，我们证明其优势源于随机梯度下降（SGD）过程中收敛至更平坦的损失曲面。大量实验验证了该方法的有效性：在Tiny-ImageNet上，相比在线KD和自KD，性能分别提升最多2.31%和3.34%。代码已开源于GitHub。
原文摘要 · Abstract (English)
Knowledge distillation (KD) is a powerful strategy for training deep neural networks (DNNs). Although it was originally proposed to train a more compact "student" model from a large "teacher" model, many recent efforts have focused on adapting it to promote generalization of the model itself, such as online KD and self KD. Here, we propose an accessible and compatible strategy named Spaced KD to improve the effectiveness of both online KD and self KD, in which the student model distills knowledge from a teacher model trained with a space interval ahead. This strategy is inspired by a prominent theory named spacing effect in biological learning and memory, positing that appropriate intervals between learning trials can significantly enhance learning performance. With both theoretical and empirical analyses, we demonstrate that the benefits of the proposed Spaced KD stem from convergence to a flatter loss landscape during stochastic gradient descent (SGD). We perform extensive experiments to validate the effectiveness of Spaced KD in improving the learning performance of DNNs (e.g., the performance gain is up to 2.31% and 3.34% on Tiny-ImageNet over online KD and self KD, respectively). Our codes have been released on github https://github.com/SunGL001/Spaced-KD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。