JumpLoRA通过动态稀疏化防止大模型持续学习中的遗忘问题。
JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models
- 用JumpReLU门控实现LoRA块的自适应稀疏，动态隔离参数
- 在IncLoRA上性能显著提升，超越当前最优方法ELLA
- 模块化设计，兼容主流LoRA类持续学习方法
基于适配器的方法已成为大语言模型持续学习的有效途径，通过为每个任务顺序学习一个低秩更新矩阵。为缓解灾难性遗忘，现有方法通过对新适配器施加约束来减少与先前适配器的干扰，或针对子空间，或针对坐标级干扰。本文提出JumpLoRA，一种通过JumpReLU门控在低秩适配（LoRA）块中自适应引入稀疏性的新框架。该方法实现动态参数隔离，有助于防止任务间干扰。我们证明该方法高度模块化，且与基于LoRA的持续学习方法兼容。具体而言，它显著提升了IncLoRA的性能，并优于当前领先的持续学习方法ELLA。
原文摘要 · Abstract (English)
Adapter-based methods have become a cost-effective approach to continual learning (CL) for Large Language Models (LLMs), by sequentially learning a low-rank update matrix for each task. To mitigate catastrophic forgetting, state-of-the-art approaches impose constraints on new adapters with respect to the previous ones, by targeting either subspace or coordinate-wise interference. In this paper, we propose JumpLoRA, a novel framework to adaptively induce sparsity in the Low-Rank Adaptation (LoRA) blocks through the use of JumpReLU gating. The method achieves dynamic parameter isolation, which helps prevent task interference. We demonstrate that our method is highly modular and compatible with LoRA-based CL approaches. Specifically, it significantly boosts the performance of IncLoRA and outperforms the leading state-of-the-art CL method, ELLA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。