OPLoRA让LoRA训练更快更准，接近SVD效果但内存更省。
Faster Than SVD, Smarter Than SGD: The OPLoRA Alternating Update
- 用交替最小二乘法优化LoRA，1-2步就逼近截断SVD
- 在多个任务上接近SVDLoRA性能，内存仅需3倍LoRA参数量
- 兼容动量，适合高效微调大模型的场景
低秩适配（LoRA）通过在冻结权重上学习低秩更新来微调大模型，显著减少可训练参数和内存占用。然而，当前LoRA与基于低秩投影的完整训练（SVDLoRA）之间仍存在差距，表明优化步骤尚有提升空间。本文提出OPLoRA，一种内存高效的优化器，将LoRA优化转化为可解释的子问题，并通过交替最小二乘法高效求解；实证发现仅需1-2次交替步骤即可接近截断SVD效果，且无需构造完整矩阵。我们还证明近期提出的预处理方法是OPLoRA的特例。OPLoRA通过相同子程序（LoRSum）维持低秩估计以支持动量，内存预算为洛拉参数数量的3倍（与Adam相当）。此外提出使用K-FAC度量的实验缩放变体。在线性任务、MNIST、CIFAR-100和RoBERTa-base（MNLI）上，OPLoRA始终接近SVDLoRA表现，同时显著降低内存消耗。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights, dramatically reducing trainable parameters and memory. However, there is still a gap between full training with low-rank projections (SVDLoRA) and LoRA fine-tuning, indicating that LoRA steps can be further improved. In this study, we propose OPLoRA, a memory-efficient optimizer that closes this gap by casting LoRA optimization as an interpretable sub-problem and solving it efficiently with alternating least squares updates, where 1-2 alternating steps are empirically found to be sufficient to closely match truncated SVD without ever forming the full matrix. We also retrieve the recently proposed preconditioning methods for LoRA as a special case. OPLoRA supports momentum by maintaining a low-rank estimate using the same subroutine (LoRSum) for computing the step, with a memory budget of 3 times the number of LoRA parameters (i.e., same as Adam). We also propose an experimental scaled variant that uses the K-FAC metric, which could be of interest. Across a linear task, MNIST, CIFAR-100, and RoBERTa-base (MNLI), OPLoRA consistently approaches SVDLoRA's performance using significantly less memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。