通过注意力机制融合多个LoRA，缓解大模型持续学习中的遗忘问题。
Learning Attentional Mixture of LoRAs for Language Model Continual Learning
- 用注意力机制动态融合多个LoRA，自适应选择任务相关参数。
- 引入L1正则使注意力向量稀疏，减少参数干扰，提升稳定性。
- 适合需要持续学习新任务的大模型应用，如多任务对话系统。
使用低秩适配（LoRA）微调大型语言模型（LLM）是持续学习新任务的有效方法，但在顺序处理多个任务时常出现灾难性遗忘。为此，我们提出注意力混合LoRA（AM-LoRA），一种专为LLM设计的持续学习方法。AM-LoRA为一系列任务学习一组LoRA，并引入注意力机制作为知识混合模块，自适应整合各LoRA的信息。该机制能有效利用每个LoRA的独特贡献，同时降低它们之间相互负面影响导致的遗忘风险。此外，在训练过程中引入$L1$范数约束，使注意力向量更稀疏，促使模型倾向于选择少数高度相关的LoRA，而非整体加权所有LoRA，从而进一步减少相互干扰的影响。在持续学习基准上的实验结果表明，所提方法具有优越性能。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) with Low-Rank adaption (LoRA) is widely acknowledged as an effective approach for continual learning for new tasks. However, it often suffers from catastrophic forgetting when dealing with multiple tasks sequentially. To this end, we propose Attentional Mixture of LoRAs (AM-LoRA), a continual learning approach tailored for LLMs. Specifically, AM-LoRA learns a sequence of LoRAs for a series of tasks to continually learn knowledge from different tasks. The key of our approach is that we devise an attention mechanism as a knowledge mixture module to adaptively integrate information from each LoRA. With the attention mechanism, AM-LoRA can efficiently leverage the distinctive contributions of each LoRA, while mitigating the risk of mutually negative interactions among them that may lead to catastrophic forgetting. Moreover, we further introduce $L1$ norm in the learning process to make the attention vector more sparse. The sparse constraints can enable the model to lean towards selecting a few highly relevant LoRAs, rather than aggregating and weighting all LoRAs collectively, which can further reduce the impact stemming from mutual interference. Experimental results on continual learning benchmarks indicate the superiority of our proposed method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。