LoRA微调时遗忘旧知识？用拉普拉斯正则化来保护模型记忆。
Mitigating Forgetting in Low Rank Adaptation
- 在LoRA权重空间引入拉普拉斯近似，估计参数置信度并限制高曲率方向更新。
- 在Llama数学推理任务中，显著提升学习与遗忘的平衡，调节正则强度即可控制效果。
- 仅对低秩部分正则化，保持轻量高效，适合资源受限场景下的持续学习。
参数高效微调方法（如低秩适配，LoRA）能快速将大模型适配到下游任务，但常导致模型丢失原有领域知识。本文提出LaLoRA，一种基于权重空间正则化的技术，通过拉普拉斯近似对LoRA进行优化。该方法估计每个参数的置信度，约束高曲率方向的更新，从而在保留先验知识的同时实现高效的特定领域学习。由于仅对LoRA权重应用拉普拉斯近似，方法保持轻量。我们在Llama模型上进行数学推理微调实验，验证了其在学习-遗忘权衡上的改进，且可通过正则强度直接调控。进一步研究了不同损失曲率近似方式、用于拉普拉斯估计的数据影响及超参数鲁棒性。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), enable fast specialization of large pre-trained models to different downstream applications. However, this process often leads to catastrophic forgetting of the model's prior domain knowledge. We address this issue with LaLoRA, a weight-space regularization technique that applies a Laplace approximation to Low-Rank Adaptation. Our approach estimates the model's confidence in each parameter and constrains updates in high-curvature directions, preserving prior knowledge while enabling efficient target-domain learning. By applying the Laplace approximation only to the LoRA weights, the method remains lightweight. We evaluate LaLoRA by fine-tuning a Llama model for mathematical reasoning and demonstrate an improved learning-forgetting trade-off, which can be directly controlled via the method's regularization strength. We further explore different loss landscape curvature approximations for estimating parameter confidence, analyze the effect of the data used for the Laplace approximation, and study robustness across hyperparameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。