低精度量化时，训练后量化为何效果差？
Understanding the Difficulty of Low-Precision Post-Training Quantization for LLMs
- 对比局部误差最小化与全局损失优化的差异
- 极低精度下，训练后量化性能显著劣于微调方法
- 适合关注大模型压缩效率的研究者
高参数量的大语言模型计算成本高昂，可通过将权重压缩至极低数值精度来提升效率。这一目标可通过训练后量化（最小化局部层内量化误差）或量化感知微调（最小化全局损失函数）实现。本研究发现，在相同数据约束下，前者几乎总是劣于后者，尤其在极低精度时更为明显。我们进一步揭示，这源于局部优化目标与全局目标之间存在严重不匹配。该结果解释了为何仅最小化局部量化误差效果有限，并强调了直接进行量化感知微调的重要性，尤其在大模型与极低精度场景下。
原文摘要 · Abstract (English)
Large language models of high parameter counts are computationally expensive, yet can be made much more efficient by compressing their weights to very low numerical precision. This can be achieved either through post-training quantization by minimizing local, layer-wise quantization errors, or through quantization-aware fine-tuning by minimizing the global loss function. In this study, we discovered that, under the same data constraint, the former approach nearly always fared worse than the latter, a phenomenon particularly prominent when the numerical precision is very low. We further showed that this difficulty of post-training quantization arose from stark misalignment between optimization of the local and global objective functions. Our findings explains limited utility in minimization of local quantization error and the importance of direct quantization-aware fine-tuning, in the regime of large models at very low precision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。