改进LoRA初始化，让微调更有效
Optimizing Fine-Tuning through Advanced Initialization Strategies for Low-Rank Adaptation
- 用新初始化策略让低秩矩阵逼近原始权重
- 在多个模型和任务上性能优于传统LoRA
- 适合追求高效微调的NLP研究者
参数高效微调方法的快速发展显著提升了大语言模型的适配效率。其中，LoRA因其在效果与参数效率间的良好平衡而广受欢迎。然而，LoRA依赖于初始化两个乘积为零的低秩矩阵，这限制了其对原始模型权重的有效激活与利用，成为性能优化的潜在瓶颈。为此，我们提出一种新型初始化策略IniLoRA，使低秩矩阵的初始化值尽可能逼近原始模型权重。实验表明，IniLoRA在多种模型与任务上均优于标准LoRA。此外，我们还引入两种变体IniLoRA-α与IniLoRA-β，分别采用不同的初始化方法进一步提升性能。
原文摘要 · Abstract (English)
The rapid development of parameter-efficient fine-tuning methods has noticeably improved the efficiency of adapting large language models. Among these, LoRA has gained widespread popularity due to its strong balance of effectiveness and parameter efficiency. However, LoRA relies on initializing two low-rank matrices whose product is zero, which limits its ability to effectively activate and leverage the original model weights-creating a potential bottleneck for optimal performance. To address this limitation, we propose \textbf{IniLoRA}, a novel initialization strategy that initializes the low-rank matrices to closely approximate the original model weights. Experimental results indicate that IniLoRA achieves better performance than LoRA across a range of models and tasks. Additionally, we introduce two variants, IniLoRA-$α$ and IniLoRA-$β$, both leveraging distinct initialization methods to enhance performance further.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。