改进量化训练中的取整与截断近似，提升大模型压缩后性能。
Continuous Approximations for Improving Quantization Aware Training of LLMs
- 用连续函数替代传统取整和截断操作，优化梯度传播。
- 量化模型在WikiText-v2上困惑度降至9.0815，优于基线9.9621。
- 适用于追求低功耗、高效率大模型部署的工程场景。
模型压缩技术用于降低大语言模型（LLMs）的计算与能耗。量化感知训练（QAT）是一种有效减少量化后性能下降的方法。为进一步缓解该问题，本文对量化过程中的取整函数和截断函数引入两种连续近似，取代传统的直通估计器（STE）。实验表明，在相同精度、模型规模与训练设置下，该方法使量化模型在WikiText-v2数据集上的困惑度（PPL）降至9.0815，优于基线的9.9621；在BoolQ任务上提升2.76%，在MMLU上提升5.47%。结果证明，该方法能更准确地学习步长与权重，推动面向可持续发展目标的节能型大模型技术发展。
原文摘要 · Abstract (English)
Model compression methods are used to reduce the computation and energy requirements for Large Language Models (LLMs). Quantization Aware Training (QAT), an effective model compression method, is proposed to reduce performance degradation after quantization. To further minimize this degradation, we introduce two continuous approximations to the QAT process on the rounding function, traditionally approximated by the Straight-Through Estimator (STE), and the clamping function. By applying both methods, the perplexity (PPL) on the WikiText-v2 dataset of the quantized model reaches 9.0815, outperforming 9.9621 by the baseline. Also, we achieve a 2.76% improvement on BoolQ, and a 5.47% improvement on MMLU, proving that the step sizes and weights can be learned more accurately with our approach. Our method achieves better performance with the same precision, model size, and training setup, contributing to the development of more energy-efficient LLMs technology that aligns with global sustainability goals.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。