OPLoRA通过正交投影防止大模型微调时的灾难性遗忘
OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuning
- 将微调更新限制在主奇异方向的正交补空间内
- 在多个任务上显著降低遗忘率，保持强任务性能
- 适合需要长期知识保留的高效微调场景
低秩适配（LoRA）虽能高效微调大语言模型，但在学习更新时会干扰编码关键预训练知识的主奇异方向，导致灾难性遗忘。本文提出正交投影LoRA（OPLoRA），通过双侧正交投影确保更新完全位于前k个主奇异子空间的正交补空间中。基于SVD分解冻结权重，使用投影 $P_L = I - U_k U_k^ op$ 与 $P_R = I - V_k V_k^ op$ 实现约束。理论证明该设计可精确保留前k个奇异三元组，提供知识保留的数学保障。为量化子空间干扰，引入 $ρ_k$ 指标衡量更新与主导方向的对齐程度。在常识推理、数学和代码生成任务上对LLaMA-2 7B与Qwen2.5 7B的实验表明，OPLoRA显著减少遗忘，同时保持优异的任务性能，验证正交投影是参数高效微调中知识保护的有效机制。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) enables efficient fine-tuning of large language models but suffers from catastrophic forgetting when learned updates interfere with the dominant singular directions that encode essential pre-trained knowledge. We propose Orthogonal Projection LoRA (OPLoRA), a theoretically grounded approach that prevents this interference through double-sided orthogonal projections. By decomposing frozen weights via SVD, OPLoRA constrains LoRA updates to lie entirely within the orthogonal complement of the top-$k$ singular subspace using projections $P_L = I - U_k U_k^\top$ and $P_R = I - V_k V_k^\top$. We prove that this construction exactly preserves the top-$k$ singular triples, providing mathematical guarantees for knowledge retention. To quantify subspace interference, we introduce $ρ_k$, a metric measuring update alignment with dominant directions. Extensive experiments across commonsense reasoning, mathematics, and code generation demonstrate that OPLoRA significantly reduces forgetting while maintaining competitive task-specific performance on LLaMA-2 7B and Qwen2.5 7B, establishing orthogonal projection as an effective mechanism for knowledge preservation in parameter-efficient fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。