提出CLoRA方法，缓解大模型持续学习中的灾难性遗忘问题。
Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models
- 在LoRA结构上引入子空间正则化，控制参数更新方向。
- 实验表明能显著降低遗忘率，性能优于传统微调方法。
- 适合需要高效微调且避免遗忘的场景，如多任务持续学习。
大语言模型在自然语言处理中表现卓越，但在学习新任务时面临灾难性遗忘问题，即适应新领域会导致旧任务性能显著下降。本文提出受控低秩适配（CLoRA），一种基于LoRA结构的子空间正则化方法。通过约束更新矩阵零空间的方向，在最小限制模型容量的前提下，减少输出变化幅度。一阶段微调与持续学习设置下的实验结果表明，CLoRA作为高效的参数高效微调方法，在缓解灾难性遗忘方面具有显著优势。对模型参数的进一步分析显示，CLoRA有效平衡了模型容量与遗忘程度之间的权衡。
原文摘要 · Abstract (English)
Large language models (LLMs) exhibit remarkable capabilities in natural language processing but face catastrophic forgetting when learning new tasks, where adaptation to a new domain leads to a substantial decline in performance on previous tasks. In this paper, we propose Controlled LoRA (CLoRA), a sub-space regularization method on LoRA structure. Aiming to reduce the scale of output change while introduce minimal constraint on model capacity, CLoRA imposes constraint on the direction of updating matrix's null space. Experimental results on one-stage LLM finetuning tasks and continual learning settings highlight the superority of CLoRA as a effective parameter efficient finetuning method with catastrophic forgetting mitigating.Further investigation for model parameters indicates that CLoRA effectively balances the trade-off between model capacity and degree of forgetting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。