用稀疏正交参数缓解持续学习中的遗忘问题。
Sparse Orthogonal Parameters Tuning for Continual Learning
- 通过稀疏正交参数融合多任务知识，避免模型更新
- 在多个持续学习基准上表现优异，无需复杂分类器
- 即插即用，适合快速部署到现有预训练模型中
基于预训练模型（PTM）的持续学习方法近年来受到关注，能够在不发生灾难性遗忘的前提下适应一系列下游任务。这类方法通常不更新预训练参数，而是引入额外的适配器、提示词和分类器。本文从新视角探讨稀疏正交参数在持续学习中的优势，发现将多个流式任务所学模型的稀疏正交性进行融合，具有极大潜力解决灾难性遗忘问题。基于此，我们提出一种新颖且高效的 SoTU（Sparse Orthogonal Parameters TUning）方法。我们假设 SoTU 的有效性源于将多领域知识转化为正交增量参数的融合。在多种持续学习基准上的实验评估表明该方法有效。值得注意的是，SoTU 在无需复杂分类器设计的情况下实现了对流式数据的最优特征表示，是一种即插即用的解决方案。
原文摘要 · Abstract (English)
Continual learning methods based on pre-trained models (PTM) have recently gained attention which adapt to successive downstream tasks without catastrophic forgetting. These methods typically refrain from updating the pre-trained parameters and instead employ additional adapters, prompts, and classifiers. In this paper, we from a novel perspective investigate the benefit of sparse orthogonal parameters for continual learning. We found that merging sparse orthogonality of models learned from multiple streaming tasks has great potential in addressing catastrophic forgetting. Leveraging this insight, we propose a novel yet effective method called SoTU (Sparse Orthogonal Parameters TUning). We hypothesize that the effectiveness of SoTU lies in the transformation of knowledge learned from multiple domains into the fusion of orthogonal delta parameters. Experimental evaluations on diverse CL benchmarks demonstrate the effectiveness of the proposed approach. Notably, SoTU achieves optimal feature representation for streaming data without necessitating complex classifier designs, making it a Plug-and-Play solution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。