改进LoRA优化理论，显著降低收敛所需计算量。
On the Convergence of Stochastic Low-Rank Adaptation
- 提出新分析方法，将确定性场景下收敛复杂度从指数级降至ε⁻⁴。
- 在随机梯度下，通过算法优化使收敛复杂度为ε⁻⁸。
- 适用于需要高效微调大模型的研究者，尤其关注低秩适配的性能保障。
低秩适配（LoRA）通过优化两个适配器矩阵 $B \in \mathbb{R}^{m \times r}$ 与 $A \in \mathbb{R}^{r \times n}$，对冻结的预训练权重矩阵 $W_\mathrm{base} \in \mathbb{R}^{m \times n}$ 进行低秩更新。已有分析表明，确定性梯度下降（LoRA-GD）需 $\exp\{\mathcal{O}(ε^{-2})\}$ 次查询才能找到满足 $\|\nabla J(B,A)\| \leq ε$ 的 $ε$-驻点。本文改进分析，证明 $\mathcal{O}(ε^{-4})$ 次全梯度评估即足够。进一步研究基于无偏梯度估计与有限方差的随机优化场景，提出 LoRA-NSGDM，实现 $\mathcal{O}(ε^{-8})$ 次随机查询即可达到相同精度。在额外均方光滑性假设下，引入方差缩减策略，提出 LoRA-STORM，将随机查询复杂度进一步降低至 $\mathcal{O}(ε^{-6})$。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) optimizes $J(B,A)=\mathcal L(W_\mathrm{base}+sBA)$ over two adapters $B \in \mathbb{R}^{m \times r}$ and $A \in \mathbb{R}^{r \times n}$ that form a low-rank update to a frozen pretrained weight matrix $W_\mathrm{base} \in \mathbb{R}^{m \times n}$. The prior analysis shows LoRA-GD takes $\exp\{\mathcal{O}(ε^{-2})\}$ oracle calls to find an $ε$-stationary point such that $\|\nabla J(B,A)\|\leq ε$ in the deterministic setting. We sharpen the analysis and show that $\mathcal{O}(ε^{-4})$ full-gradient evaluations suffice for the same first-order criterion. We further study stochastic LoRA under unbiased gradient estimates and finite variance. We propose LoRA-NSGDM, which finds an $ε$-stationary point with $\mathcal{O}(ε^{-8})$ stochastic oracle complexity. Under the additional mean-square smoothness condition, we use variance reduction strategy and propose LoRA-STORM, which improves the stochastic oracle complexity to $\mathcal{O}(ε^{-6})$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。