提出RCP方法，实现大模型2比特权值4比特激活的极致压缩。
Rotate, Clip, and Partition: Towards W2A4KV4 Quantization by Integrating Rotation and Learnable Non-uniform Quantizer
- 融合旋转与可学习非均匀量化，直接优化权值分布。
- 在LLaMA-2-7B上实现2.84点WikiText2困惑度损失，内存减少5.29倍。
- 适用于移动端及领域专用模型，训练稳定无崩溃问题。
我们提出旋转、截断与分块（RCP）方法，首次实现大语言模型在W2A4KV4（2比特权值、4比特激活、4比特键值缓存）配置下的极致压缩。RCP将近期旋转技术与新型非均匀权值量化器结合，通过定量分析随机旋转对2比特权值量化的影响。其权值量化器采用可学习直接分段（LDP），引入可学习参数，联合学习非均匀量化区间与模型权重。我们还设计了支持非均匀W2A4的专用GPU内核。实验表明，RCP可将LLaMA-2-7B压缩至W2A4KV4，仅产生2.84的WikiText2困惑度增量，内存占用降低5.29倍。此外，RCP能稳定量化移动设备目标的LLaMA-3.2以及领域专用的WizardCoder-7B和MetaMath-7B，未出现收敛失败或重复等问题。代码已开源：https://github.com/songsm921/RCP。
原文摘要 · Abstract (English)
We propose Rotate, Clip, and Partition (RCP), a quantization-aware training (QAT) approach that first realizes extreme compression of LLMs with W2A4KV4(2-bit weight, 4-bit activation, and 4-bit KV cache) configuration. RCP integrates recent rotation techniques with a novel non-uniform weight quantizer design, by quantitatively analyzing the impact of random rotation on 2-bit weight quantization. Our weight quantizer features Learnable Direct Partitioning (LDP), which introduces learnable parameters to directly learn non-uniform intervals jointly with LLM weights. We also present a specialized GPU kernel that supports GEMV on non-uniform W2A4. Experiments show that RCP can compress LLaMA-2-7B to W2A4KV4 with a loss of only 2.84 WikiText2 ppl and 5.29 times reduced memory footprint. Furthermore, RCP can quantize challenging mobile-targeted LLaMA-3.2 models and domain-specific WizardCoder-7B and MetaMath-7B with no critical problems such as convergence failure and repetition. Code is available at https://github.com/ songsm921/RCP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。