对比4种量化方法在昇腾芯片上推理大模型的性能，发现8位量化最稳，4位易出错。
A Case Study of Selected PTQ Baselines for Reasoning LLMs on Ascend NPU
- 测试了AWQ、GPTQ等4种量化算法在昇腾NPU上的表现
- 4位权值激活量化导致长文本推理逻辑崩溃，8位更稳定
- 适合关注昇腾平台部署的工程师和量化研究者
后训练量化（PTQ）对高效模型部署至关重要，但在昇腾NPU上的效果相比GPU仍缺乏研究。本文针对面向推理的模型（如DeepSeek-R1-Distill-Qwen系列1.5B/7B/14B及QwQ-32B），评估了四种代表性PTQ算法：AWQ、GPTQ、SmoothQuant与FlatQuant，涵盖从纯权重量化到旋转式高级方法。实验证明存在显著平台敏感性：4位纯权重量化在大模型上可行，但激进的4位权值-激活量化在昇腾NPU上出现逐层校准不稳，导致长上下文推理任务逻辑崩溃；而标准8位量化则保持数值稳定。此外，真实场景下的INT8部署显示，虽优化内核降低延迟，但动态量化开销仍限制端到端加速。这些结果为量化推理模型在昇腾NPU上的可行性与局限性提供了实用参考。
原文摘要 · Abstract (English)
Post-Training Quantization (PTQ) is crucial for efficient model deployment, yet its effectiveness on Ascend NPU remains under-explored compared to GPU architectures. This paper presents a case study of representative PTQ baselines applied to reasoning-oriented models such as DeepSeek-R1-Distill-Qwen series (1.5B/7B/14B) and QwQ-32B. We evaluate four distinct algorithms, including AWQ, GPTQ, SmoothQuant, and FlatQuant, to cover the spectrum from weight-only compression to advanced rotation-based methods. Our empirical results reveal significant platform sensitivity. While 4-bit weight-only quantization proves viable for larger models, aggressive 4-bit weight-activation schemes suffer from layer-wise calibration instability on the NPU, leading to logic collapse in long-context reasoning tasks. Conversely, standard 8-bit quantization remains numerically stable. Furthermore, a real-world INT8 deployment demonstrates that although optimized kernels reduce latency, dynamic quantization overheads currently limit end-to-end acceleration. These findings offer a practical reference for the feasibility and limitations of deploying quantized reasoning models on Ascend NPU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。