通过校准初始化提升量化大模型的微调效果
CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initialization
- 用小样本数据集校准量化模型，优化每层LoRA初始值
- 在极低比特（如4位）下仍保持性能领先
- 适合资源受限场景下的量化大模型微调
使用低秩适配（LoRA）微调大语言模型已成为下游任务中高效的方法，尤其在计算资源有限的情况下。然而，将LoRA应用于量化后的语言模型时，由于量化权重的表示精度降低，带来了独特挑战。本文提出CLoQ（针对量化大模型的校准LoRA初始化），一种简洁的初始化策略，旨在最小化原始模型与量化后模型在各层间的差异。通过一个小规模校准数据集，CLoQ对预训练模型进行量化，并为每一层确定最优的LoRA组件，从而为后续微调奠定坚实基础。本工作的一个关键贡献是提出了一种新理论结果，可实现最优LoRA组件的精确闭式构造。我们在语言生成、算术推理和常识推理等多个任务上验证了CLoQ的有效性，结果表明其在超低比特宽度下始终优于现有方法。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) using low-rank adaptation (LoRA) has become a highly efficient approach for downstream tasks, particularly in scenarios with limited computational resources. However, applying LoRA techniques to quantized LLMs poses unique challenges due to the reduced representational precision of quantized weights. In this paper, we introduce CLoQ (Calibrated LoRA initialization for Quantized LLMs), a simplistic initialization strategy designed to overcome these challenges. Our approach focuses on minimizing the layer-wise discrepancy between the original LLM and its quantized counterpart with LoRA components during initialization. By leveraging a small calibration dataset, CLoQ quantizes a pre-trained LLM and determines the optimal LoRA components for each layer, ensuring a strong foundation for subsequent fine-tuning. A key contribution of this work is a novel theoretical result that enables the accurate and closed-form construction of these optimal LoRA components. We validate the efficacy of CLoQ across multiple tasks such as language generation, arithmetic reasoning, and commonsense reasoning, demonstrating that it consistently outperforms existing LoRA fine-tuning methods for quantized LLMs, especially at ultra low-bit widths.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。