1-bit量化下用K均值方法提升大模型性能
1-Bit Wonder: Improving QAT Performance in the Low-Bit Regime through K-Means Quantization
- 用K均值聚类实现权重量化，优于传统整数格式
- 固定内存预算下，1比特量化在生成任务表现最佳
- 适合资源受限场景下的大模型部署优化
量化感知训练（QAT）能显著降低大语言模型的内存占用，同时保持可接受的性能损失。然而，量化格式与位宽的最优选择仍不明确，且量化与下游性能之间的权衡尚未充分理解，现有比较多依赖困惑度评估。本文通过实证研究低比特范围内的QAT，发现基于K均值的权重量化方法优于整数格式，且可在标准硬件上高效实现。进一步发现，在固定推理内存预算下，1比特量化权重在生成类下游任务中表现最优。
原文摘要 · Abstract (English)
Quantization-aware training (QAT) is an effective method to drastically reduce the memory footprint of LLMs while keeping performance degradation at an acceptable level. However, the optimal choice of quantization format and bit-width presents a challenge in practice. The full design space of quantization is not fully explored in the context of QAT, and the precise trade-off between quantization and downstream performance is poorly understood, as comparisons often rely solely on perplexity-based evaluations. In this work, we address these shortcomings with an empirical study of QAT in the low-bit regime. We show that k-means based weight quantization outperforms integer formats and can be implemented efficiently on standard hardware. Furthermore, we find that, under a fixed inference memory budget, the best performance on generative downstream tasks is achieved with $1$-bit quantized weights.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。