改进低秩微调方法，让大模型训练更快更稳。
RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models
- 每步动态选择最优低秩分解，平衡权重更新
- 在多个大模型上收敛速度提升，性能超越现有方法
- 几乎无额外计算开销，适合高效微调场景
低秩适应（LoRA）通过更新预训练权重矩阵的低维子空间来降低大模型微调的计算与内存开销。尽管高效，但因低秩分解不唯一导致权重更新不一致且失衡，造成收敛不佳和性能下降。本文提出重构低秩适应（RefLoRA），每步选取最小化损失上界的最优低秩分解，使损失曲面更平坦，实现一致且均衡的权重更新，从而加速稳定收敛。在DeBERTaV3、LLaMA-7B、LLaMA2-7B和LLaMA3-8B等主流大语言模型上，对自然语言理解与常识推理任务进行广泛实验。数值结果表明，RefLoRA收敛更快，优于多种基准，且相比先进LoRA变体计算开销可忽略不计。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) lowers the computational and memory overhead of fine-tuning large models by updating a low-dimensional subspace of the pre-trained weight matrix. Albeit efficient, LoRA exhibits suboptimal convergence and noticeable performance degradation, due to inconsistent and imbalanced weight updates induced by its nonunique low-rank factorizations. To overcome these limitations, this article identifies the optimal low-rank factorization per step that minimizes an upper bound on the loss. The resultant refactored low-rank adaptation (RefLoRA) method promotes a flatter loss landscape, along with consistent and balanced weight updates, thus speeding up stable convergence. Extensive experiments evaluate RefLoRA on natural language understanding, and commonsense reasoning tasks with popular large language models including DeBERTaV3, LLaMA-7B, LLaMA2-7B and LLaMA3-8B. The numerical tests corroborate that RefLoRA converges faster, outperforms various benchmarks, and enjoys negligible computational overhead compared to state-of-the-art LoRA variants.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。