动态提示变压器缓解遗忘，提升持续学习性能
DPFormer: Dynamic Prompt Transformer for Continual Learning
- 用可变提示机制保存旧知识并适应新任务
- 在多个数据集上达到当前最佳准确率
- 适合需要长期学习新知识的场景
持续学习中，解决灾难性遗忘常陷入稳定与灵活性的矛盾。此外，因任务间缺乏知识交互，还会出现任务混淆问题。为此，我们提出一种新型动态提示变压器（DPFormer），通过提示机制帮助模型记忆先前类别和任务的知识，并在单一网络结构下以几乎固定参数量持续学习新知识。该机制还能为不同任务提供差异信息，缓解任务混淆。基于提示设计，我们构建了统一分类模块，结合二元交叉熵损失、知识蒸馏损失和辅助损失，实现端到端训练。在不同类增量设置下，我们的方法在CIFAR-100、ImageNet100和ImageNet1K数据集上均取得最优表现。代码将在论文录用后开源。
原文摘要 · Abstract (English)
In continual learning, solving the catastrophic forgetting problem may make the models fall into the stability-plasticity dilemma. Moreover, inter-task confusion will also occur due to the lack of knowledge exchanges between different tasks. In order to solve the aforementioned problems, we propose a novel dynamic prompt transformer (DPFormer) with prompt schemes. The prompt schemes help the DPFormer memorize learned knowledge of previous classes and tasks, and keep on learning new knowledge from new classes and tasks under a single network structure with a nearly fixed number of model parameters. Moreover, they also provide discrepant information to represent different tasks to solve the inter-task confusion problem. Based on prompt schemes, a unified classification module with the binary cross entropy loss, the knowledge distillation loss and the auxiliary loss is proposed to train the whole model in an end-to-end trainable manner. Compared with state-of-the-art methods, our method achieves the best performance in the CIFAR-100, ImageNet100 and ImageNet1K datasets under different class-incremental settings in continual learning. The source code will be available at our GitHub after acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。