3-bit量化让大模型性能接近全精度，突破现有压缩瓶颈
SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantization
- 用贪心算法优化各通道比特分配，提升量化效率
- 引入可训练缩放向量，实现任意位数下的高性能量化
- 适合追求极致压缩比的部署场景，尤其在低比特下表现优异
大型语言模型在各类任务中表现出色，但推理部署面临资源消耗大的挑战。现有量化方法在低精度下性能下降明显，且通常仅支持有限比特位数，需大量人工调参。为此，我们提出SKIM：一种基于缩放K均值聚类的混合精度量化方法。该方法包含两项创新：1）采用贪心算法近似求解权重通道的最优比特分配；2）引入可训练缩放向量，处理非可微的K均值聚类。该方法可适配任意比特数，显著提升性能。在模型困惑度上，3比特量化后的LLaMA模型与全精度版本的差距平均缩小了16.3%。
原文摘要 · Abstract (English)
Large Language Models (LLMs) exhibit impressive performance across various tasks, but deploying them for inference poses challenges. Their high resource demands often necessitate complex, costly multi-GPU pipelines, or the use of smaller, less capable models. While quantization offers a promising solution utilizing lower precision for model storage, existing methods frequently experience significant performance drops at lower precision levels. Additionally, they typically provide only a limited set of solutions at specific bit levels, many of which are extensively manually tuned. To address these challenges, we propose a new method called SKIM: Scaled K-means clustering wIth Mixed precision. Our approach introduces two novel techniques: 1. A greedy algorithm to solve approximately optimal bit allocation across weight channels, and 2. A trainable scaling vector for non-differentiable K-means clustering. These techniques substantially improve performance and can be adapted to any given bit. Notably, in terms of model perplexity, our method narrows the gap between 3-bit quantized LLaMA models and their full precision counterparts by 16.3% on average.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。