通过旋转量化提升推理模型精度，显著降低误差积累。
ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference
- 引入成对旋转与通道缩放，均衡各通道幅度分布。
- 在权重仅量化下，推理任务平均准确率比AWQ高2.4%。
- 适合追求高效低误差部署的推理类大模型应用。
后训练量化（PTQ）将大语言模型（LLM）的权重和激活值压缩为低精度表示，以减少内存占用并加速推理。然而，权重和激活值中的异常值常导致较大量化误差，尤其在近期推理型LLM中，误差会沿长推理链累积。现有PTQ方法要么无法充分抑制异常值，要么推理时引入显著开销。本文提出成对旋转量化（ParoQuant），结合硬件高效的独立Givens旋转与通道级缩放，使各通道幅度趋于均衡，缩小每组量化内的动态范围，有效缓解异常值问题。我们还协同设计了推理内核，充分利用GPU并行性，确保运行时旋转与缩放轻量。在仅权重量化条件下，ParoQuant在推理任务上平均比AWQ提升2.4%准确率，开销低于10%。其性能亦可媲美最先进的权值-激活联合量化方法，为推理型LLM的高效高精度部署开辟新路径。
原文摘要 · Abstract (English)
Post-training quantization (PTQ) compresses the weights and activations of large language models (LLMs) into low-precision representations to reduce memory footprint and accelerate inference. However, the presence of outliers in weights and activations often leads to large quantization errors and severe accuracy degradation, especially in recent reasoning LLMs where errors accumulate across long chains of thought. Existing PTQ methods either fail to sufficiently suppress outliers or introduce significant overhead during inference. In this paper, we propose Pairwise Rotation Quantization (ParoQuant), a PTQ method that combines hardware-efficient and optimizable independent Givens rotations with channel-wise scaling to even out the magnitudes across channels and narrow the dynamic range within each quantization group, effectively addressing the outlier issue. We further co-design the inference kernel to fully exploit GPU parallelism and keep the rotations and scaling lightweight at runtime. Under weight-only quantization, ParoQuant achieves an average 2.4% accuracy improvement over AWQ on reasoning tasks, with less than 10% overhead. ParoQuant also matches the accuracy of state-of-the-art weight-activation quantization methods. This paves the way for more efficient and accurate deployment of reasoning LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。