针对持续学习中旧类重复出现的现实场景，提出双策略提升模型稳定性与泛化能力。
Multi-Level Knowledge Distillation and Dynamic Self-Supervised Learning for Continual Learning
- 多层级知识蒸馏融合多阶段模型特征与输出，保留更全面的历史知识。
- 动态自监督学习利用外部无标签数据加速新类学习，权重自适应避免干扰主任务。
- 在CVPR 2024持续学习挑战赛中取得第二名，适合实际部署的持续学习系统。
类增量学习中带重复（CIR）场景比传统设定更贴近现实，因每次任务均会重新引入已学类别。该设定假设可轻松获取大量外部无标签数据（如互联网数据）。为此，本文提出两个组件：多层级知识蒸馏（MLKD），从多个历史模型中跨层次、多角度（特征与logits）提取知识，增强模型对过往知识的保持；动态自监督学习（SSL），利用无标签数据加速新类学习，同时通过动态加权机制确保训练聚焦主任务。两项技术显著提升模型在CIR设置下的性能，在CVPR第5届持续学习挑战赛中获第二名。
原文摘要 · Abstract (English)
Class-incremental with repetition (CIR), where previously trained classes repeatedly introduced in future tasks, is a more realistic scenario than the traditional class incremental setup, which assumes that each task contains unseen classes. CIR assumes that we can easily access abundant unlabeled data from external sources, such as the Internet. Therefore, we propose two components that efficiently use the unlabeled data to ensure the high stability and the plasticity of models trained in CIR setup. First, we introduce multi-level knowledge distillation (MLKD) that distills knowledge from multiple previous models across multiple perspectives, including features and logits, so the model can maintain much various previous knowledge. Moreover, we implement dynamic self-supervised loss (SSL) to utilize the unlabeled data that accelerates the learning of new classes, while dynamic weighting of SSL keeps the focus of training to the primary task. Both of our proposed components significantly improve the performance in CIR setup, achieving 2nd place in the CVPR 5th CLVISION Challenge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。