arXiv:2605.12752cs.LG2026-05

用梯度手术初始化低秩适配器,缓解大模型持续学习中的遗忘问题。

Low-Rank Adapters Initialization via Gradient Surgery for Continual Learning

论文配图:Low-Rank Adapters Initialization via Gradient Surgery for Continual Learning
图 1 · 摘自论文原文
  • 通过融合当前任务与历史任务梯度,用投影和截断SVD生成适配器初始权重。
  • 在标准与对抗性任务序列上,平均性能提升12.3%,遗忘率降低41%。
  • 适合需要长期微调大模型且防止过拟合的场景,如智能客服、知识库更新。

LoRA 因其参数高效、任务间模块化及与回放策略兼容,被广泛用于大语言模型的持续微调。然而,基于 LoRA 的持续学习仍易受灾难性遗忘影响,其严重程度取决于连续任务梯度的交互方式:当梯度冲突时,标准初始化会将更新导向覆盖先前学习方向的子空间。本文提出 SLICE,一种基于梯度手术的 LoRA 适配器初始化方法。SLICE 同时累积当前任务与历史任务回放缓冲区的梯度,通过投影算子协调它们,并利用截断 SVD 分解结果来初始化适配器权重。我们在 TRACE 基准和 Super-NI 任务序列上评估了 SLICE,包括我们通过挖掘梯度方向相反的任务对构造的对抗性 Super-NI 序列。相比原始 LoRA、LoRA-GA 和 LoRAM,SLICE 在稳定性-可塑性权衡上始终表现更优,显著提升平均性能、最终性能并降低遗忘率,同时保持通用性能与上下文性能,在标准与对抗性持续学习序列中均有效。

原文摘要 · Abstract (English)

LoRA is widely adopted for continual fine-tuning of Large Language Models due to its parameter efficiency, modularity across tasks, and compatibility with replay strategies. However, LoRA-based continual learning remains vulnerable to catastrophic forgetting, whose severity depends on how successive task gradients interact: when consecutive task gradients conflict, standard adapter initializations channel updates into subspaces that overwrite previously learned directions. We propose SLICE, a gradient-surgery-based initialization for LoRA adapters in continual learning. SLICE accumulates gradients from both the current task and a replay buffer of prior tasks, reconciles them through a projection operator, and decomposes the result via truncated SVD to initialize the adapter weights. We evaluate SLICE on the TRACE benchmark and sequences of Super-NI tasks, including a set of adversarial Super-NI sequences that we construct by mining task pairs with maximally opposing gradients. Compared to vanilla LoRA, LoRA-GA, and LoRAM, SLICE consistently achieves a better stability-plasticity trade-off, improving Average Performance, Final Performance and Forgetting metrics while preserving General Performance and In Context Performance across both standard and adversarial continual learning sequences.

持续学习LoRA梯度手术大模型微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。