LoRS让稀疏大模型微调更省内存和计算。
LoRS: Efficient Low-Rank Adaptation for Sparse Large Language Model
- 通过权重重计算与计算图重排，保持稀疏性同时降低开销。
- 在稀疏LLM微调中，内存与计算消耗显著下降。
- 适合追求高效微调的模型压缩与部署场景。
现有低秩适配（LoRA）方法在稀疏大语言模型（LLMs）上面临难以维持稀疏性的挑战。近期工作通过引入额外掩码机制改进了稀疏性保持，但导致内存与计算开销增加，影响效率。为此，我们提出LoRS，一种在微调稀疏LLM时兼具内存与计算效率的新方法。该方法采用权重重计算与计算图重排策略，有效缓解稀疏性维护带来的资源压力。此外，通过优化适配器初始化，进一步提升性能。实验表明，LoRS在保持高性能的同时，显著降低了微调阶段的内存与计算消耗，优于现有LoRA方法。
原文摘要 · Abstract (English)
Existing low-rank adaptation (LoRA) methods face challenges on sparse large language models (LLMs) due to the inability to maintain sparsity. Recent works introduced methods that maintain sparsity by augmenting LoRA techniques with additional masking mechanisms. Despite these successes, such approaches suffer from an increased memory and computation overhead, which affects efficiency of LoRA methods. In response to this limitation, we introduce LoRS, an innovative method designed to achieve both memory and computation efficiency when fine-tuning sparse LLMs. To mitigate the substantial memory and computation demands associated with preserving sparsity, our approach incorporates strategies of weight recompute and computational graph rearrangement. In addition, we also improve the effectiveness of LoRS through better adapter initialization. These innovations lead to a notable reduction in memory and computation consumption during the fine-tuning phase, all while achieving performance levels that outperform existing LoRA approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。