用乘法更新保持参数几何结构，解决大模型持续学习遗忘问题
Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Models
- 基于李群的乘法微调，保持参数内在几何结构
- 在标准持续学习基准上表现最优，支持大规模任务序列
- 无需记忆回放或任务编号，适合真实场景部署
大型语言模型在顺序多任务学习中面临灾难性遗忘问题。现有参数正则化方法（如 O-LoRA、N-LoRA）通过低秩子空间正交性缓解干扰，但加法更新会扭曲模型参数的内在几何结构。我们提出 extbf{OLieRA}，一种基于李群的微调框架，通过乘法更新保持参数几何结构，并在不同任务子空间间强制正交性。OLieRA 在标准持续学习基准上达到当前最优性能，且在长任务序列下仍保持高竞争力。它继承了 O-LoRA 的无回放和无任务ID推理特性，为大语言模型的持续学习建立了一个理论完备的新范式。
原文摘要 · Abstract (English)
Large language models (LLMs) suffer from catastrophic forgetting in sequential multi-task learning. Existing parameter regularization methods (e.g., O-LoRA, N-LoRA) mitigate interference via low-rank subspace orthogonality, but additive updates distort the intrinsic geometry of model parameters. We propose \textbf{OLieRA}, a Lie group based fine-tuning framework that preserves parameter geometry through multiplicative updates while enforcing orthogonality across task subspaces. OLieRA achieves state-of-the-art performance on the Standard CL benchmark and remains highly competitive under large task sequences. It further inherits the replay-free and task-ID free inference properties of O-LoRA, establishing a principled paradigm for continual learning in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。