用动态静态提示协同机制,让模型在少量样本下持续学习新类别不遗忘旧知识。
DSS-Prompt: Dynamic-Static Synergistic Prompting for Few-Shot Class-Incremental Learning
- 引入动态与静态提示协同,增强预训练模型对新类别的适应能力。
- 在四个基准上均超越现有方法,且有效缓解灾难性遗忘问题。
- 无需微调即可实现强性能,适合快速部署的增量学习场景。
近期基于大规模预训练模型的学习在众多下游任务中取得显著成功,但在少样本类增量学习(FSCIL)这一挑战性任务中仍研究不足。该任务旨在仅用少量样本持续学习新概念,同时避免遗忘旧知识。本文提出DSS-Prompt,一种简单而有效的方案,通过在预训练Vision Transformer中以提示方式最小修改,将其转化为强大的FSCIL分类器。具体地,在每个Transformer块中协同使用两类互补提示:静态提示用于弥合预训练数据与下游数据之间的领域差异,提升适配性;动态提示用于捕捉实例级语义,实现从基础类到新类的轻松迁移。特别地,动态提示通过预训练多模态模型提取输入相关多样化语义生成,并自适应调整其在不同层的重要性。在此基础上,仅需一个简单的原型分类器,即可在不进行增量任务微调的情况下超越当前最优结果。我们在四个基准上进行了广泛实验,验证了DSS-Prompt的有效性,结果显示其在所有数据集上均优于现有方法,并能有效缓解灾难性遗忘问题。
原文摘要 · Abstract (English)
Learning from large-scale pre-trained models with strong generalization ability has shown remarkable success in a wide range of downstream tasks recently, but it is still underexplored in the challenging few-shot class-incremental learning (FSCIL) task. It aims to continually learn new concepts from limited training samples without forgetting the old ones at the same time. In this paper, we introduce DSS-Prompt, a simple yet effective approach that transforms the pre-trained Vision Transformer with minimal modifications in the way of prompts into a strong FSCIL classifier. Concretely, we synergistically utilize two complementary types of prompts in each Transformer block: static prompts to bridge the domain gap between the pre-training and downstream datasets, thus enabling better adaption; and dynamic prompts to capture instance-aware semantics, thus enabling easy transfer from base to novel classes. Specially, to generate dynamic prompts, we leverage a pre-trained multi-modal model to extract input-related diverse semantics, thereby generating complementary input-aware prompts, and then adaptively adjust their importance across different layers. In this way, on top of the prompted visual embeddings, a simple prototype classifier can beat state-of-the-arts without further training on the incremental tasks. We conduct extensive experiments on four benchmarks to validate the effectiveness of our DSS-Prompt and show that it consistently achieves better performance than existing approaches on all datasets and can alleviate the catastrophic forgetting issue as well.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。