通过高秩预热优化LoRA初始化方向，提升微调效果
HRP: High-Rank Preheating for Superior LoRA Initialization
- 用高秩预热获取近似最优初始方向
- 实验显示性能优于随机初始化，且随预热秩增大持续提升
- 适合追求高效微调的模型开发者
本文研究低秩适应（LoRA）中初始化的关键影响。理论分析表明，LoRA微调结果对初始化高度敏感，可能导致次优低秩结果。尽管可通过调整初始方向至目标ΔW的主要奇异向量来缓解此问题，但该方向在实际中通常未知。为此，我们提出高秩预热（HRP）：先以较高预热秩训练几轮，再利用得到的BA⊤主奇异向量作为主微调的初始化。仅修改初始方向，即证明HRP在期望下优于随机初始化，且提升随预热秩增加而增强。大量实验证明，HRP显著提升LoRA有效性，在多种模型与任务中超越其他初始化策略及LoRA变体。
原文摘要 · Abstract (English)
This paper studies the crucial impact of initialization in Low-Rank Adaptation (LoRA). Through theoretical analysis, we demonstrate that the fine-tuned result of LoRA is highly sensitive to initialization, which is likely to lead suboptimal low-rank results. While this issue can be mitigated by adjusting the initial direction towards the main singular vectors of the target $ΔW$, which is, however, typically unknown in real-world scenarios. To approximate this initial direction, we propose High-Rank Preheating (HRP), which first trains LoRA with a higher preheating rank for a few steps, then uses the main singular vectors of the derived $BA^\top$ as initialization for the main fine-tuning process. With only a modification in the initial direction, we prove that HRP makes LoRA achieve better fine-tuned results than random initialization in expectation, and the enhancement grows with the preheating rank. We validate our theoretical findings through extensive experiments in various models and tasks, where HRP significantly enhances LoRA's effectiveness and outperforms other initialization strategies and other LoRA variants.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。