4位权值+8位浮点计算，大幅提速且保持精度
Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference
- 用4位整数存权重，8位浮点做计算，利用硬件优势
- 相比16位运算，吞吐量提升明显，精度损失可控
- 无需额外计算开销，适合现代加速器部署
深度神经网络在自然语言处理、计算机视觉和语音识别等任务中取得了顶尖性能。但随着任务复杂度上升，模型规模持续增大，带来延迟和内存效率挑战。为应对这些限制，后训练量化成为有效方案。本文提出一种新型硬件高效量化与推理方法，利用硬件特性实现最小精度损失。具体地，提出W4A8方案：权重采用4位整数量化存储，推理计算使用8位浮点数执行，相比16位操作显著提升速度并改善内存利用率，适用于多种现代加速器。为缓解精度下降，设计了名为双精度量化（DPQ）的新算法，利用该方案的结构特点，在不增加推理开销的前提下实现高精度。实验表明，该方法在保持可接受精度损失的同时，显著提升了吞吐量。
原文摘要 · Abstract (English)
Deep neural networks have achieved state-of-the-art results in a wide range of applications, from natural language processing and computer vision to speech recognition. However, as tasks become increasingly complex, model sizes continue to grow, posing challenges in latency and memory efficiency. To meet these constraints, post-training quantization has emerged as a promising solution. In this paper, we propose a novel hardware-efficient quantization and inference scheme that exploits hardware advantages with minimal accuracy degradation. Specifically, we introduce a W4A8 scheme, where weights are quantized and stored using 4-bit integer precision, and inference computations are performed using 8-bit floating-point arithmetic, demonstrating significant speedups and improved memory utilization compared to 16-bit operations, applicable on various modern accelerators. To mitigate accuracy loss, we develop a novel quantization algorithm, dubbed Dual Precision Quantization (DPQ), that leverages the unique structure of our scheme without introducing additional inference overhead. Experimental results demonstrate improved performance (i.e., increased throughput) while maintaining tolerable accuracy degradation relative to the full-precision model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。