基于旋转不变性，精准识别重要通道实现高效量化。
ROSAQ: Rotation-based Saliency-Aware Weight Quantization for Efficiently Compressing Large Language Models
- 在投影特征空间中通过PCA找重要通道，而非原始空间。
- 混合精度量化：关键通道用FP16，其余用INT3/4，提升压缩效率。
- 生成256词元时速度比FP16快2.3倍，适合部署大型语言模型。
量化是降低大语言模型内存需求的有效技术，可能同时提升延迟性能。利用Transformer的旋转不变性特性，我们提出旋转式显著性感知权重量化（ROSAQ），在投影特征空间中识别显著通道，而非原始特征空间。其中，投影后的主成分维度被视为自然显著特征。ROSAQ包含三部分：1）基于PCA的投影，对校准集进行主成分分析并经由PCA投影变换；2）显著通道识别，选取对应前K个最大特征值的维度作为显著通道；3）显著性感知的混合精度量化，对显著维度使用FP16，其余维度使用INT3/4。实验表明，ROSAQ在原始特征空间的显著性感知量化及其他现有量化方法上均表现更优。结合内核融合后，在批量大小为64的情况下生成256个词元时，相比FP16实现约2.3倍的速度提升。
原文摘要 · Abstract (English)
Quantization has been widely studied as an effective technique for reducing the memory requirement of large language models (LLMs), potentially improving the latency time as well. Utilizing the characteristic of rotational invariance of transformer, we propose the rotation-based saliency-aware weight quantization (ROSAQ), which identifies salient channels in the projection feature space, not in the original feature space, where the projected "principal" dimensions are naturally considered as "salient" features. The proposed ROSAQ consists of 1) PCA-based projection, which first performs principal component analysis (PCA) on a calibration set and transforms via the PCA projection, 2) Salient channel dentification, which selects dimensions corresponding to the K-largest eigenvalues as salient channels, and 3) Saliency-aware quantization with mixed-precision, which uses FP16 for salient dimensions and INT3/4 for other dimensions. Experiment results show that ROSAQ shows improvements over the baseline saliency-aware quantization on the original feature space and other existing quantization methods. With kernel fusion, ROSAQ presents about 2.3x speed up over FP16 implementation in generating 256 tokens with a batch size of 64.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。