提出新型量化框架,用图模型优化低比特部署,显著降低显存占用和推理延迟。
Minimizing the Hidden Cost of Scales: Graph-Guided Ultra-Low-Bit Quantization for Large Language Models

- 通过分布统计分离重要与不重要权重,构建稀疏图估计每层最优分组数
- 双模式量化:重要权重多比特保留,不重要权重二值化,平均仅需1.03位权重+0.004位缩放
- 自适应敏感度阈值选择,适合大规模语言模型高效推理部署
后训练量化(PTQ)对大语言模型高效部署至关重要。现有超低比特PTQ方法依赖刚性权重重要性假设或位置启发式,带来显著隐藏的缩放开销。本文提出SAGE-PTQ(基于重要性的图引导高效PTQ),一种新型超低比特量化框架,旨在最小化隐藏缩放成本。SAGE-PTQ利用分布统计分离重要与不重要权重,并将采样后的不重要权重建模为稀疏图,以估算每层最优分组数。采用双模式量化策略:对重要权重分配多比特精度,对不重要权重进行二值化。为减少缩放开销,仅对重要权重使用每通道一个缩放参数,对不重要组使用单一标量。最后,通过自适应敏感度阈值选择每矩阵最优重要性比例。SAGE-PTQ在平均上实现1.03权重比特和0.004缩放比特/矩阵,优于当前最佳方法如BiLLM和PB-LLM。在LLaMA-3-8B上,其维基文本2困惑度为6.74,远低于BiLLM的55.8,且显存使用不足其50%。在LLaMA-2-70B上,单块NVIDIA L40 GPU下推理速度提升1.5倍,验证了实际推理效率。
原文摘要 · Abstract (English)
Post-training quantization (PTQ) is critical for the efficient deployment of large language models (LLMs). Recent ultra-low-bit PTQ methods rely on rigid weight-saliency assumptions or position heuristics, introducing substantial hidden scaling overhead. We propose SAGE-PTQ (Saliency-Aware Graph-guided Efficient PTQ), a novel ultra-low-bit quantization framework for LLMs that minimizes hidden scaling cost. SAGE-PTQ separates salient and unsalient weights using distributional statistics, then models subsampled unsalient weights as a sparse graph to estimate the optimal number of groups per layer. SAGE-PTQ applies dual-mode quantization, assigning multi-bit precision to salient weights and binarizing unsalient weights. To reduce scaling overhead, SAGE-PTQ uses one per-channel scale for salient weights and one scalar per unsalient group. Finally, SAGE-PTQ implements adaptive saliency thresholding to select the optimal saliency ratio per matrix. SAGE-PTQ achieves 1.03 weight bits and only 0.004 scaling bits per matrix on average, outperforming state-of-the-art methods such as BiLLM and PB-LLM. On LLaMA-3-8B, SAGE-PTQ achieves 6.74 WikiText2 perplexity, compared to 55.8 for BiLLM, while using less than 50% of BiLLM's GPU memory. On LLaMA-2-70B, SAGE-PTQ provides 1.5x faster decoding on one NVIDIA L40 GPU, demonstrating practical inference efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。