PoLoRA优化LoRA微调，提速1.2-1.7倍且更稳定。
PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer
- 设计矩阵与乘积结构感知的更新方向，结合曲率预处理和大小控制规则。
- 在1B到8B模型上，比Adam少1.2-1.7倍步数达相同损失，每步开销≤3%。
- 对学习率不敏感，最优学习率随秩变化稳定，适合高效微调场景。
低秩适配（LoRA）通过在权重矩阵中添加可训练的低秩更新来降低大语言模型微调成本，通常使用Adam优化器训练。但传统Adam忽略参数的矩阵及乘积结构。尽管已有如Muon等矩阵感知优化器，其表现并不一致优于Adam，且近期提出的乘积感知变体也未带来稳定提升。为此，本文提出PoLoRA——一种基于三个核心成分的预条件正交化LoRA优化器：乘积感知的谱更新方向、基于控制单样本损失变化的曲率预条件，以及控制因子与合并更新大小的幅度规则。在代码与数学指令微调数据集上评估，覆盖1B至8B参数模型，结果表明PoLoRA可在1.2至1.7倍更少步数内达到与调优后的Adam相当的最终损失，每步额外开销不超过3%。相比Adam，PoLoRA对学习率不敏感，最优学习率在不同秩下保持稳定。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) makes finetuning large language models cheaper by adding to each weight matrix a trainable low-rank update parameterized as the product of two matrices. These matrices are usually trained with Adam, which treats them as a single flat vector of parameters and ignores both the matrix and product structure of LoRA. Applying a matrix-aware optimizer such as Muon to each factor does not consistently improve over Adam, and neither do the product-aware Muon variants proposed in concurrent works. To realize consistent gains, we introduce PoLoRA, a Preconditioned Orthogonalized LoRA optimizer built from three ingredients: a product-aware spectral update direction, curvature preconditioning derived from controlling the per-sample loss change, and a magnitude rule that controls the sizes of both the factor and merged updates. We evaluate PoLoRA on instruction-tuning datasets for code and math across models from 1B to 8B parameters, and find that it reaches the final held-out loss achieved by tuned Adam in 1.2-1.7 times fewer steps, while adding at most 3% per-step overhead. Compared to Adam, PoLoRA is also less sensitive to the learning rate, and its optimal learning rate is stable across ranks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。