非零初始化让LoRA更抗学习率波动，不需从预训练模型开始微调。
Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
- 同时非零初始化LoRA的A、B矩阵，提升微调稳定性。
- 非零初始化虽引入随机噪声，但不影响最终性能。
- 适合追求稳定微调、减少调参成本的研究者。
低秩适应（LoRA）是一种广泛使用的参数高效微调方法。标准LoRA层中，矩阵A或B被初始化为零，以确保微调从预训练模型开始，但这一做法缺乏理论支持。本文从无限宽度视角研究了非零初始化对LoRA微调动态的影响。分析表明，与零初始化相比，同时将A和B初始化为非零值能显著提升LoRA对次优学习率（尤其是较小学习率）的鲁棒性。进一步分析显示，尽管非零初始化会向预训练权重引入随机噪声，但通常不会影响微调性能。这意味着微调无需严格从预训练模型出发。实验在多种模型和数据集上验证了结论的有效性。代码已开源：https://github.com/Leopold1423/non_zero_lora-icml25。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method. In standard LoRA layers, one of the matrices, $A$ or $B$, is initialized to zero, ensuring that fine-tuning starts from the pretrained model. However, there is no theoretical support for this practice. In this paper, we investigate the impact of non-zero initialization on LoRA's fine-tuning dynamics from an infinite-width perspective. Our analysis reveals that, compared to zero initialization, simultaneously initializing $A$ and $B$ to non-zero values improves LoRA's robustness to suboptimal learning rates, particularly smaller ones. Further analysis indicates that although the non-zero initialization of $AB$ introduces random noise into the pretrained weight, it generally does not affect fine-tuning performance. In other words, fine-tuning does not need to strictly start from the pretrained model. The validity of our findings is confirmed through extensive experiments across various models and datasets. The code is available at https://github.com/Leopold1423/non_zero_lora-icml25.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。