LoRA-Mini通过分解矩阵，将可训练参数减少20倍。
LoRA-Mini : Adaptation Matrices Decomposition and Selective Training
- 将低秩矩阵拆分为四部分，仅训练其中两个内层矩阵。
- 可训练参数量比标准LoRA减少最多20倍，性能相近。
- 适合资源受限场景下的大模型高效微调，如边缘设备。
大语言模型（LLMs）的快速发展推动了自然语言处理的进步，也带来了对高效、任务特定微调方法的需求。传统微调需更新大量参数，计算成本高且内存占用大。低秩适配（LoRA）通过减少可训练参数实现参数高效微调。然而，LoRA模块仍带来显著存储负担。我们提出LoRA-Mini，通过将低秩矩阵分解为四部分，仅训练其中两个内层矩阵，实现可训练参数数量相比标准LoRA最高降低20倍，同时保持与标准LoRA相当的性能水平，有效兼顾计算与存储效率。
原文摘要 · Abstract (English)
The rapid advancements in large language models (LLMs) have revolutionized natural language processing, creating an increased need for efficient, task-specific fine-tuning methods. Traditional fine-tuning of LLMs involves updating a large number of parameters, which is computationally expensive and memory-intensive. Low-Rank Adaptation (LoRA) has emerged as a promising solution, enabling parameter-efficient fine-tuning by reducing the number of trainable parameters. However, while LoRA reduces the number of trainable parameters, LoRA modules still create significant storage challenges. We propose LoRA-Mini, an optimized adaptation of LoRA that improves parameter efficiency by splitting low-rank matrices into four parts, with only the two inner matrices being trainable. This approach achieves upto a 20x reduction compared to standard LoRA in the number of trainable parameters while preserving performance levels comparable to standard LoRA, addressing both computational and storage efficiency in LLM fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。