调中间主成分可更好平衡模型性能与遗忘，提升微调稳定性。
Least but not Last: Fine-tuning Intermediate Principal Components for Better Performance-Forgetting Trade-Offs
- 用权重矩阵主成分初始化LoRA，选中间组件而非首尾
- 在多任务中实现高精度且显著减少知识遗忘
- 适合追求稳定微调的开发者和研究者
低秩适应（LoRA）方法已成为在计算和内存受限下将大模型适配到下游任务的关键技术。然而，现有方法在任务性能提升与预训练知识灾难性遗忘之间难以平衡，且推荐策略不一致。本文系统分析了以权重矩阵主成分作为初始化时低秩适应中的性能-遗忘权衡问题。研究发现，微调中间主成分相比现有方法中的首（PiSSA）或尾（MiLoRA）组件，能带来更优的平衡性和对高学习率的鲁棒性。基于此，我们提出了实用的LoRA初始化指南，可在多种计算机视觉与自然语言处理任务上实现高准确率并显著降低遗忘。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) methods have emerged as crucial techniques for adapting large pre-trained models to downstream tasks under computational and memory constraints. However, they face a fundamental challenge in balancing task-specific performance gains against catastrophic forgetting of pre-trained knowledge, where existing methods provide inconsistent recommendations. This paper presents a comprehensive analysis of the performance-forgetting trade-offs inherent in low-rank adaptation using principal components of weight matrices as initialization. Our investigation reveals that fine-tuning intermediate components leads to better balance and robustness to high learning rates than first (PiSSA) and last (MiLoRA) components in existing work. Building on these findings, we provide practical guidelines for initialization of LoRA methods to balance the performance-forgetting trade-off. In a thorough empirical study on a variety of computer vision and NLP tasks we confirm that these guidelines achieve high accuracy and reduced forgetting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。