通过动态稀疏性加速大模型微调，兼顾速度与精度
SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
- 基于轻量级奇异值分解的稀疏性估计器，动态选择关键权重
- 计算成本降低2.2倍，实测速度提升1.6倍，准确率不变
- 适合需要高效微调的大模型应用，如代码生成与推理任务
大模型微调计算与内存开销巨大。尽管参数高效微调方法（如QLoRA、DoRA）减少了可训练参数并降低内存占用，但未降低计算成本，甚至可能拖慢训练。本文提出SparseLoRA，通过上下文稀疏性加速微调。设计了一种无需训练的轻量级SVD稀疏性估计算法，动态选取用于损失与梯度计算的稀疏权重子集。系统分析并解决了层、词元和训练步之间的敏感性问题。实验表明，SparseLoRA在各类下游任务（包括常识推理、算术推理、代码生成、指令遵循）中，计算成本最多降低2.2倍，实测速度提升最高达1.6倍，且保持原有准确率。
原文摘要 · Abstract (English)
Fine-tuning LLMs is both computationally and memory-intensive. While parameter-efficient fine-tuning methods, such as QLoRA and DoRA, reduce the number of trainable parameters and lower memory usage, they do not decrease computational cost. In some cases, they may even slow down fine-tuning. In this paper, we introduce SparseLoRA, a method that accelerates LLM fine-tuning through contextual sparsity. We propose a lightweight, training-free SVD sparsity estimator that dynamically selects a sparse subset of weights for loss and gradient computation. Also, we systematically analyze and address sensitivity across layers, tokens, and training steps. Our experimental results show that SparseLoRA reduces computational cost by up to 2.2 times and a measured speedup of up to 1.6 times while maintaining accuracy across various downstream tasks, including commonsense and arithmetic reasoning, code generation, and instruction following.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。