CE-LoRA让大模型微调更快，算力消耗更低。
CE-LoRA: Computation-Efficient LoRA Fine-Tuning for Language Models
- 用稀疏计算替代全矩阵乘法，聚焦关键参数
- 双LoRA设计降低梯度误差传播，提速30%以上
- 适合资源有限但需高效微调的大模型应用
大型语言模型在各类任务中表现优异，但微调仍需大量计算资源。尽管低秩适应（LoRA）显著降低了显存占用，但对计算成本的削减有限。本文识别出激活梯度计算是LoRA反向传播的主要瓶颈，提出计算高效的LoRA（CE-LoRA）算法，在保持内存效率的同时提升计算效率。该方法采用两项关键技术：近似矩阵乘法，将大规模稠密乘法替换为仅涉及关键行与列的稀疏乘法；双LoRA机制，减少激活梯度中的误差传播。理论上，CE-LoRA收敛速度与LoRA一致，为$\mathcal{O}(1/\sqrt{T})$，其中$T$为迭代次数。实验表明，相比LoRA，CE-LoRA在不明显损失性能的前提下显著降低计算开销。
原文摘要 · Abstract (English)
Large Language Models (LLMs) demonstrate exceptional performance across various tasks but demand substantial computational resources even for fine-tuning computation. Although Low-Rank Adaptation (LoRA) significantly alleviates memory consumption during fine-tuning, its impact on computational cost reduction is limited. This paper identifies the computation of activation gradients as the primary bottleneck in LoRA's backward propagation and introduces the Computation-Efficient LoRA (CE-LoRA) algorithm, which enhances computational efficiency while preserving memory efficiency. CE-LoRA leverages two key techniques: Approximated Matrix Multiplication, which replaces dense multiplications of large and complete matrices with sparse multiplications involving only critical rows and columns, and the Double-LoRA technique, which reduces error propagation in activation gradients. Theoretically, CE-LoRA converges at the same rate as LoRA, $ \mathcal{O}(1/\sqrt{T}) $, where $T$ is the number of iteartions. Empirical evaluations confirm that CE-LoRA significantly reduces computational costs compared to LoRA without notable performance degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。