仅训练1%参数，实现高效低比特大模型量化。
DL-QAT: Weight-Decomposed Low-Rank Quantization-Aware Training for Large Language Models
- 将权重分解为低秩矩阵，仅更新极小部分参数。
- 3比特下比现有方法在MMLU上高4.2%准确率。
- 适合资源受限场景下的大模型部署优化。
提升大语言模型推理效率是研究热点。后训练量化（PTQ）在低比特时易出问题，而量化感知训练（QAT）虽有效但计算开销大。为此，我们提出权重分解的低秩量化感知训练（DL-QAT），仅训练总参数量的不足1%。通过引入分组量化幅度调节每组缩放，并在组内使用LoRA矩阵调整量化空间中的权重大小与方向。在LLaMA和LLaMA2模型上验证，结果显著优于基线方法。例如，对3比特的LLaMA-7B模型，其在MMLU上的表现比之前最优方法高出4.2%。同时，预训练模型的量化结果也超越现有QAT方法，证明本方法兼具优异性能与高效率。
原文摘要 · Abstract (English)
Improving the efficiency of inference in Large Language Models (LLMs) is a critical area of research. Post-training Quantization (PTQ) is a popular technique, but it often faces challenges at low-bit levels, particularly in downstream tasks. Quantization-aware Training (QAT) can alleviate this problem, but it requires significantly more computational resources. To tackle this, we introduced Weight-Decomposed Low-Rank Quantization-Aware Training (DL-QAT), which merges the advantages of QAT while training only less than 1% of the total parameters. Specifically, we introduce a group-specific quantization magnitude to adjust the overall scale of each quantization group. Within each quantization group, we use LoRA matrices to update the weight size and direction in the quantization space. We validated the effectiveness of our method on the LLaMA and LLaMA2 model families. The results show significant improvements over our baseline method across different quantization granularities. For instance, for LLaMA-7B, our approach outperforms the previous state-of-the-art method by 4.2% in MMLU on 3-bit LLaMA-7B model. Additionally, our quantization results on pre-trained models also surpass previous QAT methods, demonstrating the superior performance and efficiency of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。