从梯度流视角解析LoRA学习动态，揭示初始化对微调效果的关键影响。
Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorization
- 通过梯度流分析矩阵分解中的LoRA，强调初始值的重要性。
- 小初始化使误差更小，且能改善预训练模型与目标矩阵的奇异空间对齐。
- 提出谱初始化方法，理论上可实现任意精度的微调，适合关注理论机制的研究者。
尽管低秩适应（LoRA）在微调预训练模型中表现出色，但对其一阶优化方法与精心设计的初始化如何适配新任务仍缺乏理论理解。本文首次从梯度流（GF）角度分析矩阵分解（MF）中的LoRA学习动态，强调初始化的关键作用。对于小初始化，理论证明梯度流收敛至最优解附近，且初始值越小，最终误差越低。分析显示，最终误差受预训练模型与目标矩阵奇异空间错位影响，减小初始化尺度可提升对齐程度。为此，我们提出一种谱初始化方法，并证明在小谱初始化下，梯度流可任意精度收敛到微调任务。矩阵分解与图像分类的数值实验验证了上述结论。
原文摘要 · Abstract (English)
Despite the empirical success of Low-Rank Adaptation (LoRA) in fine-tuning pre-trained models, there is little theoretical understanding of how first-order methods with carefully crafted initialization adapt models to new tasks. In this work, we take the first step towards bridging this gap by theoretically analyzing the learning dynamics of LoRA for matrix factorization (MF) under gradient flow (GF), emphasizing the crucial role of initialization. For small initialization, we theoretically show that GF converges to a neighborhood of the optimal solution, with smaller initialization leading to lower final error. Our analysis shows that the final error is affected by the misalignment between the singular spaces of the pre-trained model and the target matrix, and reducing the initialization scale improves alignment. To address this misalignment, we propose a spectral initialization for LoRA in MF and theoretically prove that GF with small spectral initialization converges to the fine-tuning task with arbitrary precision. Numerical experiments from MF and image classification validate our findings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。