首次证明LoRA梯度下降以1/log T速率收敛，解决其理论空白。
On the Convergence Rate of LoRA Gradient Descent
- 通过重参数化适配器矩阵外积，构建非渐近收敛分析框架。
- 在无Lipschitz光滑性假设下，证明收敛速率为O(1/log T)。
- 适用于关注大模型微调理论保障的研究者与工程实践者。
低秩适配（LoRA）因其卓越性能和低计算开销，近年在大模型微调中广受欢迎。LoRA通过训练两个“适配器”矩阵，以低秩形式表示模型参数，大幅减少每步更新的参数量。尽管方法简单，其收敛性因缺乏Lipschitz光滑性而难以分析，现有理论仅限于渐近行为或强有界性假设。本文首次对原始LoRA梯度下降算法进行非渐近收敛分析，无需此类假设。研究基于三个关键步骤：(i) 将问题重述为堆叠适配器矩阵的外积；(ii) 构建针对“类Lipschitz”重参数化函数的修正下降引理；(iii) 控制步长。由此证明，LoRA梯度下降以速率O(1/log T)收敛至驻点，其中T为迭代次数。数值实验验证了理论结果。
原文摘要 · Abstract (English)
The low-rank adaptation (LoRA) algorithm for fine-tuning large models has grown popular in recent years due to its remarkable performance and low computational requirements. LoRA trains two ``adapter" matrices that form a low-rank representation of the model parameters, thereby massively reducing the number of parameters that need to be updated at every step. Although LoRA is simple, its convergence is poorly understood due to the lack of Lipschitz smoothness, a key condition for classic convergence analyses. As a result, current theoretical results only consider asymptotic behavior or assume strong boundedness conditions which artificially enforce Lipschitz smoothness. In this work, we provide for the first time a non-asymptotic convergence analysis of the \textit{original LoRA gradient descent} algorithm, which reflects widespread practice, without such assumptions. Our work relies on three key steps: i) reformulating the problem in terms of the outer product of the stacked adapter matrices, ii) a modified descent lemma for the ``Lipschitz-like" reparametrized function, and iii) controlling the step size. With this approach, we prove that LoRA gradient descent converges to a stationary point at rate $O(\frac{1}{\log T})$, where $T$ is the number of iterations. We conduct numerical experiments to validate our theoretical findings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。