通过自蒸馏与错分样本优先存储，缓解在线增量学习中的灾难性遗忘。
Reducing Catastrophic Forgetting in Online Class Incremental Learning Using Self-Distillation
- 用浅层输出作教师，通过自蒸馏提取可迁移知识。
- 在CIFAR10/100和MiniImageNet上优于传统方法。
- 适合资源受限下持续学习大量类别的场景。
持续学习中存在灾难性遗忘问题，即模型在学习新任务时会遗忘旧知识。现有重放方法虽有效，但在有限内存缓冲区下泛化能力受限。本文通过自蒸馏方式，利用浅层具有强泛化性的输出作为教师，提取可迁移知识。针对大规模类别或复杂数据带来的收敛困难与过拟合风险，提出一种新的记忆更新策略，优先存储易被误分类的样本。在CIFAR10、CIFAR100和MiniImageNet数据集上的实验表明，该方法显著优于传统方法。
原文摘要 · Abstract (English)
In continual learning, there is a serious problem of catastrophic forgetting, in which previous knowledge is forgotten when a model learns new tasks. Various methods have been proposed to solve this problem. Replay methods which replay data from previous tasks in later training, have shown good accuracy. However, replay methods have a generalizability problem from a limited memory buffer. In this paper, we tried to solve this problem by acquiring transferable knowledge through self-distillation using highly generalizable output in shallow layer as a teacher. Furthermore, when we deal with a large number of classes or challenging data, there is a risk of learning not converging and not experiencing overfitting. Therefore, we attempted to achieve more efficient and thorough learning by prioritizing the storage of easily misclassified samples through a new method of memory update. We confirmed that our proposed method outperformed conventional methods by experiments on CIFAR10, CIFAR100, and MiniimageNet datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。