LoRA提速不稳，新方法实现更稳定高效微调
LoRA Is Slower Than You Think
- 分析LoRA在不同模型架构下的速度差异，定位性能瓶颈
- 提出新方法，在多种设置下实现更一致的训练加速
- 适合资源受限场景下的LLM微调实践者参考
低秩适应（LoRA）是微调大语言模型最广泛使用的技术之一。通过引入少量可训练的低秩权重矩阵，LoRA显著减少了需更新的参数量，相比全量微调在内存消耗和计算效率上具有明显优势。然而，我们发现LoRA在不同模型架构和训练配置下并非始终带来速度提升。基于此不一致性，本文对LoRA性能进行了全面分析，探究其速度提升受限的根本原因。根据研究结果，提出了几种更高效的LLM微调方法。通过实证评估，验证了所提方法在性能相当或更优的同时，能提供更一致的训练加速效果。本工作为资源受限条件下优化大模型微调提供了重要洞察与实用指导。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) is one of the most widely used techniques for fine-tuning large language models (LLMs). By introducing a small number of trainable low-rank weight matrices, LoRA substantially reduces the number of parameters that need to be updated, offering significant advantages in memory consumption and computational efficiency compared to full fine-tuning. However, we observed that LoRA does not consistently provide speed improvements across all model architectures and training setups. Motivated by this inconsistency, we conduct a comprehensive analysis of LoRA's performance and investigate the underlying factors limiting its speedup. Based on our findings, we propose several methods for more efficient fine-tuning of LLMs. We empirically evaluate these methods and compare them to LoRA, demonstrating that our approach achieves comparable or superior performance while delivering more consistent training speed improvements. Our work offers valuable insights and practical guidelines for practitioners seeking to optimize LLM fine-tuning under resource constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。