用图神经网络动态分配位宽，提升低比特大模型量化效果
Mixed-Precision Graph Neural Quantization for Low Bit Large Language Models
- 通过图神经网络捕捉权重间依赖关系，自适应分配不同位宽
- 在<3比特下超越GPTQ，WikiText2和C4上性能创纪录
- 适合资源受限场景下的大模型部署，尤其关注低比特压缩
后训练量化（PTQ）对于在资源受限环境下部署大语言模型至关重要，可显著降低资源需求。然而，现有PTQ方法在低于3比特时性能下降明显，主要因量化权重与原始权重差异过大。为提升低比特下的量化表现，本文提出混合精度图神经网络后训练量化（MG-PTQ），引入图神经网络（GNN）模块以捕捉权重间的依赖关系，并自适应分配量化位宽。通过GNN的信息传播机制，该方法更准确评估权重重要性，优化量化策略分配。在WikiText2和C4数据集上的大量实验表明，MG-PTQ优于当前最优的GPTQ方法，在低比特条件下创下新基准。
原文摘要 · Abstract (English)
Post-Training Quantization (PTQ) is pivotal for deploying large language models (LLMs) within resource-limited settings by significantly reducing resource demands. However, existing PTQ strategies underperform at low bit levels < 3 bits due to the significant difference between the quantized and original weights. To enhance the quantization performance at low bit widths, we introduce a Mixed-precision Graph Neural PTQ (MG-PTQ) approach, employing a graph neural network (GNN) module to capture dependencies among weights and adaptively assign quantization bit-widths. Through the information propagation of the GNN module, our method more effectively captures dependencies among target weights, leading to a more accurate assessment of weight importance and optimized allocation of quantization strategies. Extensive experiments on the WikiText2 and C4 datasets demonstrate that our MG-PTQ method outperforms previous state-of-the-art PTQ method GPTQ, setting new benchmarks for quantization performance under low-bit conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。