2比特量化新方法，用少量数据就能让大模型保持高精度。
LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization
- 用线性约束向量量化替代传统量化，实现可训练的权重压缩。
- 仅需0.1%到10%训练数据，性能超越现有顶尖方法。
- 适合资源受限场景下的大模型高效部署，尤其看重低比特推理。
量化感知训练（QAT）对极低比特大型语言模型至关重要。当前主流方法基于标量量化（SQ），虽优化高效，但在2比特精度下性能严重下降。而向量量化（VQ）虽具更强表征能力，但其离散码本查找阻碍端到端训练。本文提出LC-QAT，一种2比特权重专用的VQ-QAT框架，通过学习仿射映射表示量化权重，获得高质量的权重量化初始化，并在训练前向中无需显式码本查找，实现全可微端到端优化。该强初始化使LC-QAT具备高度数据效率。跨多种LLM的实验表明，相比最先进QAT方法，LC-QAT始终表现更优，且仅需0.1%至10%训练数据。结果确立了LC-QAT作为极端低比特模型部署的实用且可扩展方案。代码已公开于https://github.com/AI9Stars/UniSVQ。
原文摘要 · Abstract (English)
Quantization-aware training (QAT) is essential for extremely low-bit large language models (LLMs). Current QAT methods are mainly based on scalar quantization (SQ), which enables efficient optimization but suffers from severe performance degradation at 2-bit precision. On the other hand, vector quantization (VQ) provides substantially higher representational capacity, but its discrete codebook lookup prevents end-to-end training. We propose LC-QAT, a 2-bit weight-only VQ-QAT framework that represents quantized weights via a learned affine mapping over discrete vectors, which yields a high-quality PTQ initialization and enables fully differentiable end-to-end optimization without explicit codebook lookup in the training forward pass. This strong post-training initialization makes LC-QAT highly data-efficient. Experiments across diverse LLMs demonstrate that LC-QAT consistently outperforms state-of-the-art QAT methods while using only 0.1%--10% of the training data. Our results establish LC-QAT as a practical and scalable solution for extreme low-bit model deployment. Codes are publicly available at https://github.com/AI9Stars/UniSVQ.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。