让模型持续学习多个教师，突破知识遗忘瓶颈
Continual Distillation of Teachers from Different Domains
- 用外部无标签数据实现跨域知识迁移
- 提出SE2D方法，减少后续学习对之前知识的遗忘
- 适合需要持续学习、跨领域泛化的场景
深度学习模型规模持续扩大，部分模型存储量超过大型数据集。为此，我们提出一种新范式——持续蒸馏（CD），学生模型可依次从教师模型流中学习，无需保留早期教师的访问权限。CD面临两大挑战：教师训练数据不可见，且教师专业能力各异。我们发现，外部无标签数据可实现未见知识迁移（UKT），使学生获取教师知晓但训练数据中未出现的领域知识；同时发现顺序蒸馏会导致未见知识遗忘（UKF），即后续学习会丢失先前获得的知识。为更好平衡UKT与UKF，我们提出自外部数据蒸馏（SE2D），通过保留外部数据上的输出概率（logits）来稳定异构教师间的知识学习。多基准测试表明，SE2D显著降低UKF并提升跨域泛化能力。代码已公开于：https://github.com/Nicolas1203/continual_distillation。
原文摘要 · Abstract (English)
Deep learning models continue to scale, with some requiring more storage than many large-scale datasets. Thus, we introduce a new paradigm: Continual Distillation (CD), where a student learns sequentially from a stream of teacher models without retaining access to earlier teachers. CD faces two challenges: teacher training data is unavailable, and teachers have varying expertise. We show that external unlabeled data enables Unseen Knowledge Transfer (UKT), allowing the student to acquire information from domains not present in the training data, while known to the teacher. We also show that sequential distillation causes Unseen Knowledge Forgetting (UKF) when transferred knowledge is lost after training on later teachers. To better trade off between UKT and UKF, we propose Self External Data Distillation (SE2D), a method that preserves logits on external data to stabilize learning across heterogeneous teachers. Experiments on multiple benchmarks show that SE2D reduces UKF and improves cross-domain generalization. The code and implementation for this work are publicly available at: https://github.com/Nicolas1203/continual_distillation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。