通过稀疏融合适配器，用更少参数实现持续学习中的稳定记忆。
Sparse Adapter Fusion for Continual Learning in NLP
- 动态决定复用、新增或空适配器，减少参数冗余。
- 在多个任务上性能接近顶尖方法，参数使用低于60%。
- 适合资源受限场景下的持续学习应用。
自然语言处理中的持续学习对适应不断变化的数据和防止灾难性遗忘至关重要。尽管已有显著进展，现有方法仍面临跨任务参数复用效率低、任务差异大时易遗忘、每任务引入新参数导致相似任务间知识共享受阻等问题。为此，我们提出稀疏适配器融合方法（SAFM），通过动态融合旧适配器与新适配器来应对这些挑战。SAFM分为决策阶段与调优阶段：在决策阶段，判断是否引入新适配器、复用已有适配器或添加空适配器；架构搜索过程优先复用或添加空适配器，以最小化参数消耗并最大化复用。在调优阶段，采用分层损失函数促进适配器间的差异化，有效捕捉同一任务内的知识。实验表明，SAFM在多个任务上持续优于当前最优方法，性能相当的同时参数用量不足60%。
原文摘要 · Abstract (English)
Continual learning in natural language processing plays a crucial role in adapting to evolving data and preventing catastrophic forgetting. Despite significant progress, existing methods still face challenges, such as inefficient parameter reuse across tasks, risking catastrophic forgetting when tasks are dissimilar, and the unnecessary introduction of new parameters for each task, which hampers knowledge sharing among similar tasks. To tackle these issues, we propose a Sparse Adapter Fusion Method (SAFM), which dynamically fuses old and new adapters to address these challenges. SAFM operates in two stages: the decision stage and the tuning stage. In the decision stage, SAFM determines whether to incorporate a new adapter, reuse an existing one, or add an empty adapter. The architecture search procedure, designed to prioritize reusing or adding empty adapters, minimizes parameter consumption and maximizes reuse. In the tuning stage, SAFM especially facilitates a layer-wise loss to encourage differentiation between adapters, effectively capturing knowledge within the same task. Experimental results consistently show that SAFM outperforms state-of-the-art (SOTA) methods, achieving comparable performance while utilizing less than 60% of the parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。