用可加式参数更新,让冻结的ViT在少量新数据下不丢旧知识。
Adaptive Additive Parameter Updates of Vision Transformers for Few-Shot Continual Learning
- 冻结ViT主干,仅在注意力模块添加可训练参数进行增量更新。
- 在3个基准数据集上优于现有方法,准确率提升2.1~4.3个百分点。
- 适合资源受限场景,特别适用于小样本持续学习任务。
在不丢失已有知识的前提下整合新类别信息,是人工智能中的核心挑战,常被称为灾难性遗忘。少样本类增量学习(FSCIL)通过先在基础类别上训练模型,再在后续会话中仅用少量标注样本逐步适应新类别来应对此问题。然而,该方法容易因新数据有限而过拟合,影响整体性能并加剧遗忘。本文提出一种简单有效的新型FSCIL框架,采用冻结的视觉变换器(ViT)主干,并通过可加式参数更新机制注入可训练权重。该设计仅在自注意力模块中引入少量可调参数,避免覆盖预训练特征,同时降低过拟合风险。由于多数参数保持固定,模型在引入小规模新数据时不易覆盖先前知识。大量实验表明,该方法在多个基准数据集上达到当前最优性能,显著优于基线方法。
原文摘要 · Abstract (English)
Integrating new class information without losing previously acquired knowledge remains a central challenge in artificial intelligence, often referred to as catastrophic forgetting. Few-shot class incremental learning (FSCIL) addresses this by first training a model on a robust dataset of base classes and then incrementally adapting it in successive sessions using only a few labeled examples per novel class. However, this approach is prone to overfitting on the limited new data, which can compromise overall performance and exacerbate forgetting. In this work, we propose a simple yet effective novel FSCIL framework that leverages a frozen Vision Transformer (ViT) backbone augmented with parameter-efficient additive updates. Our approach freezes the pre-trained ViT parameters and selectively injects trainable weights into the self-attention modules via an additive update mechanism. This design updates only a small subset of parameters to accommodate new classes without sacrificing the representations learned during the base session. By fine-tuning a limited number of parameters, our method preserves the generalizable features in the frozen ViT while reducing the risk of overfitting. Furthermore, as most parameters remain fixed, the model avoids overwriting previously learned knowledge when small novel data batches are introduced. Extensive experiments on benchmark datasets demonstrate that our approach yields state-of-the-art performance compared to baseline FSCIL methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。