提出量化感知初始化，让低精度微调模型性能更接近高精度版本。
QuAILoRA: Quantization-Aware Initialization for LoRA
- 在初始化阶段考虑量化误差，减少微调时的性能损失。
- 4比特量化下，困惑度降低75%，下游任务准确率提升86%。
- 无需额外显存，适合资源受限场景下的大模型微调。
QLoRA通过量化基础大语言模型降低了微调内存开销，但量化误差会损害微调后模型性能。本文提出QuAILoRA，一种量化感知的LoRA初始化方法，在不增加微调内存消耗的前提下，通过少量计算开销降低初始阶段的量化误差。我们在多个因果语言建模和下游任务上评估了不同规模与架构的模型，结果表明几乎所有使用QuAILoRA微调的LLM都取得了更低的验证困惑度。在下游任务中，其性能提升与量化误差负面影响成正比：平均而言，将4比特QLoRA应用到QuAILoRA，可实现相当于将量化精度提升至8比特时75%的困惑度下降和86%的准确率提升，且微调过程显存占用不变。
原文摘要 · Abstract (English)
QLoRA reduces the memory-cost of fine-tuning a large language model (LLM) with LoRA by quantizing the base LLM. However, quantization introduces quantization errors that negatively impact model performance after fine-tuning. In this paper we introduce QuAILoRA, a quantization-aware initialization for LoRA that mitigates this negative impact by decreasing quantization errors at initialization. Our method spends a small amount of computational overhead to compute this quantization-aware initialization, without increasing the memory-cost of fine-tuning. We evaluate our method on several causal language modeling and downstream evaluation tasks using several different model sizes and families. We observe that almost all LLMs fined-tuned with QuAILoRA achieve better validation perplexity. When evaluated on downstream tasks, we find that QuAILoRA yields improvements proportional to the negative effect of quantization error. On average, applying QuAILoRA to 4-bit QLoRA models yields 75% of the validation perplexity decrease and 86% of the downstream task accuracy increase as doubling the quantization precision to 8-bit, without increasing GPU memory utilization during fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。