提出统一稀疏量化格式，兼顾精度与推理速度。
SQ-format: A Unified Sparse-Quantized Hardware-friendly Data Format for LLMs
- 设计新型稀疏量化数据格式，融合压缩与加速机制。
- 在保持W4A8精度的同时，实现媲美W8A8的算力效率。
- 适合硬件加速器设计，尤其适用于存在异常值的激活值。
后训练量化（PTQ）在降低大语言模型（LLMs）部署门槛中起关键作用。然而，现有低比特量化与稀疏化技术难以在精度与效率间取得平衡，主要受限于硬件支持不足。例如，W4A8的峰值算力仅相当于W8A8，而主流GPU支持的2:4半结构化稀疏格式因精度损失较少被采用。为此，本文提出稀疏量化格式（SQ-format），一种可被新硬件和现有GPU支持的统一数据格式。该格式利用稀疏矩阵可在高精度下加速、低精度乘法也可随之加速的特性，实现性能与吞吐量的帕累托改进。特别适用于具有异常值不均衡特性的激活值，使其可进行静态压缩。实验表明，使用SQ-format达到当前最优的PTQ性能，并提出其硬件支持需求，为下一代AI加速器的设计提供探索方向与洞见。
原文摘要 · Abstract (English)
Post-training quantization (PTQ) plays a crucial role in the democratization of large language models (LLMs). However, existing low-bit quantization and sparsification techniques are difficult to balance accuracy and efficiency due to the limited hardware support. For example, W4A8 can only achieve the same peak TOPS as W8A8 whereas the GPU-supported sparse data format (2:4 semi-structure sparse) is seldomly adopted due to the loss of accuracy. To bridge this gap, in this paper, we propose the Sparse-Quantized Format (SQ-format), which is a unified data format for quantization and sparsification potentially easily supported by new hardware and existing GPUs. SQ-format makes use of the fact that sparse matrix can be accelerated in high-precision, and low-precision matrix multiplication can also be accelerated accordingly. As such, SQ-format is proposed to achieve Pareto improvement between performance and throughput. This format is particularly suitable for activations with outlier inequality status and makes their static compression possible. We show the state-of-the-art PTQ performance with SQ-format, propose the hardware required to support it, and further offer the design exploration and insights for the next-generation AI accelerators.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。