让大模型持续学习更高效且不遗忘，通过融合高低秩参数更新。
Continual Gradient Low-Rank Projection Fine-Tuning for LLMs
- 联合优化全参数与低秩参数，在统一低秩梯度空间中更新。
- 在多个持续学习基准上超越现有最优方法，有效缓解灾难性遗忘。
- 适合需要高效持续训练大模型的研究者和工业应用
大语言模型的持续微调面临效率与表达能力之间的权衡。低秩适应（LoRA）虽高效，但受限于低秩特性及显式参数约束，影响新任务学习与知识迁移。我们提出GORP（梯度低秩投影）持续学习训练策略，通过协同结合全参数与低秩参数，并在统一的低秩梯度子空间中联合更新，拓展优化空间的同时保持效率并减轻灾难性遗忘。大量实验证明，GORP在持续学习基准上优于现有最先进方法。代码已开源：https://github.com/Wcxwcxw/GORP。
原文摘要 · Abstract (English)
Continual fine-tuning of Large Language Models (LLMs) is hampered by the trade-off between efficiency and expressiveness. Low-Rank Adaptation (LoRA) offers efficiency but constrains the model's ability to learn new tasks and transfer knowledge due to its low-rank nature and reliance on explicit parameter constraints. We propose GORP (Gradient LOw Rank Projection) for Continual Learning, a novel training strategy that overcomes these limitations by synergistically combining full and low-rank parameters and jointly updating within a unified low-rank gradient subspace. GORP expands the optimization space while preserving efficiency and mitigating catastrophic forgetting. Extensive experiments on continual learning benchmarks demonstrate GORP's superior performance compared to existing state-of-the-art approaches. Code is available at https://github.com/Wcxwcxw/GORP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。