揭示LoRA在持续学习中遗忘的几何规律，解释为何低秩适配器会忘记任务。
Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation
- 用梯度子空间夹角建模遗忘，提出可量化预测遗忘程度的几何公式。
- 发现当任务子空间角度大时，遗忘与适配器秩无关，仅在相似任务下受秩影响。
- 适用于研究参数高效微调的持续学习者，尤其关注模型遗忘机制的工程师。
低秩适配（LoRA）作为高效微调大模型的方法日益流行，但其在持续学习中的行为仍不明确。本文从梯度子空间交互的角度提出一种几何理论，揭示遗忘由简单几何规律决定：$\mathcal{F} = α(1 - \cos^2θ_{\min}) + β$，其中$θ_{\min}$为任务梯度子空间间的最小主角。该公式揭示了近似秩不变性：在高子空间夹角下，遗忘基本不受适配器秩影响（合成实验中变异系数≈0.8%；真实基准上为10%-19%，表明依赖于具体场景而非绝对成立）。我们在合成任务（相关系数$r=0.994$）、Split-CIFAR100（ViT-LoRA）和顺序GLUE（RoBERTa-LoRA）上验证了该理论。分析统一了解读文献中看似矛盾的结果：仅当任务子空间相近（低角）时，秩才影响遗忘；而如O-LoRA等正交方法在自然正交性已高的情况下收益有限。这些发现为参数高效微调下的持续学习提供了理论指导。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient approach for adapting large pre-trained models, yet its behavior under continual learning remains poorly understood. We present a geometric theory characterizing catastrophic forgetting in LoRA through the lens of gradient subspace interactions. Our central finding is that forgetting is governed by a simple geometric law: $\mathcal{F} = α(1 - \cos^2θ_{\min}) + β$, where $θ_{\min}$ is the minimum principal angle between task gradient subspaces. This formulation reveals an approximate rank-invariance property, at high subspace angles, forgetting becomes largely independent of the adapter rank (coefficient of variation $\approx 0.8\%$ in controlled synthetic settings; CV $\approx 10$-$19\%$ on real benchmarks, suggesting this is regime-dependent rather than absolute). We validate our theory on synthetic tasks ($r=0.994$ correlation), Split-CIFAR100 with ViT-LoRA, and sequential GLUE with RoBERTa-LoRA. Our analysis reconciles seemingly contradictory findings in the literature: we show that rank affects forgetting only when task subspaces are similar (low angle), while orthogonal methods like O-LoRA provide minimal benefit when natural orthogonality is already high. These insights provide principled guidance for continual learning with parameter-efficient fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。