提出两步幂次量化法，让大模型低精度推理更快更准。
PoTPTQ: A Two-step Power-of-Two Post-training for LLMs
- 分两步优化量化尺度：先定起点，再用小样本微调。
- 2-3比特下精度超越现有方法，推理速度提升3.67倍。
- 专为GPU优化，解决符号位纠缠问题，适合部署在显卡上。
大语言模型在自然语言处理任务中表现优异，但部署需大量计算资源。幂次量化（PoT）可降低计算开销，但现有方法在GPU上因符号位纠缠和串行位操作导致去量化效率低。本文提出一种新型两步后训练量化框架，通过稳健的初始量化尺度与极小校准集微调，显著提升低精度（如2-3比特）下的模型精度。所提方法在浮点推理中加速去量化过程，在NVIDIA V100上实现3.67倍提速，在RTX 4090上达1.63倍，性能优于当前最优整数量化方案。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable performance across various natural language processing (NLP) tasks. However, their deployment is challenging due to the substantial computational resources required. Power-of-two (PoT) quantization is a general tool to counteract this difficulty. Albeit previous works on PoT quantization can be efficiently dequantized on CPUs using fixed-point addition, it showed less effectiveness on GPUs. The reason is entanglement of the sign bit and sequential bit manipulations needed for dequantization. We propose a novel POT quantization framework for LLM weights that (i) outperforms state-of-the-art accuracy in extremely low-precision number formats, and (ii) enables faster inference through more efficient dequantization. To maintain the accuracy of the quantized model, we introduce a two-step post-training algorithm: (i) initialize the quantization scales with a robust starting point, and (ii) refine these scales using a minimal calibration set. The performance of our PoT post-training algorithm surpasses the current state-of-the-art in integer quantization, particularly at low precisions such as 2- and 3-bit formats. Our PoT quantization accelerates the dequantization step required for the floating point inference and leads to $3.67\times$ speed up on a NVIDIA V100, and $1.63\times$ on a NVIDIA RTX 4090, compared to uniform integer dequantization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。