解决LoRA微调中权重更新受缩放旋转影响的问题，提升训练效率与性能。
LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization
- 引入矩阵预条件机制实现变换不变性，使权重更新不受因子缩放影响。
- 在Gemma-2B上替换Adam后，指令任务准确率提升4.6%，多基准平均提升3.5%。
- 适用于各类大模型微调，尤其适合追求高效低资源训练的场景。
低秩适配（LoRA）是一种广泛使用的参数高效微调方法，可降低大语言模型的内存需求。然而，现有LoRA优化器缺乏变换不变性，导致权重更新结果依赖于两个LoRA因子的缩放或旋转方式，进而造成学习效率低下和次优解。本文提出一种新型自适应矩阵预条件方法LoRA-RITE，能实现变换不变性且计算开销低。我们提供了理论分析证明该方法的优势，并在Gemma 2B、7B及mT5-XXL等模型上进行了多种大语言模型任务实验。结果表明，相较于现有优化器，本方法表现一致更优。例如，在Gemma-2B的LoRA微调中用LoRA-RITE替代Adam，Super-Natural Instructions任务准确率提升4.6%，其他四个基准（HellaSwag、ArcChallenge、GSM8K、OpenBookQA）平均提升3.5%。
原文摘要 · Abstract (English)
Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements. However, current LoRA optimizers lack transformation invariance, meaning the actual updates to the weights depends on how the two LoRA factors are scaled or rotated. This deficiency leads to inefficient learning and sub-optimal solutions in practice. This paper introduces LoRA-RITE, a novel adaptive matrix preconditioning method for LoRA optimization, which can achieve transformation invariance and remain computationally efficient. We provide theoretical analysis to demonstrate the benefit of our method and conduct experiments on various LLM tasks with different models including Gemma 2B, 7B, and mT5-XXL. The results demonstrate consistent improvements against existing optimizers. For example, replacing Adam with LoRA-RITE during LoRA fine-tuning of Gemma-2B yielded 4.6\% accuracy gain on Super-Natural Instructions and 3.5\% accuracy gain across other four LLM benchmarks (HellaSwag, ArcChallenge, GSM8K, OpenBookQA).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。