通过低秩残差保留关键激活，实现大模型4比特量化下的高精度推理。
ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
- 用主成分分析定位高方差低秩子空间，仅对其中系数保留8比特精度。
- 在多个基准上比现有方法降低33%困惑度，推理速度提升3倍。
- 适合追求极致推理效率的部署场景，尤其适用于大模型轻量化落地。
后训练量化（PTQ）有望显著降低大语言模型推理时的计算开销。将所有权重、激活和键值缓存张量统一量化至4比特，同时保持模型泛化能力极具挑战性，主要因激活中的极端异常值导致高量化误差。为此，我们提出ResQ，一种突破当前技术极限的PTQ方法。通过主成分分析（PCA），识别出激活方差最高的低秩子空间（实际为隐藏维度的1/8），并在此子空间内保留高精度（如8比特）系数，其余部分量化至4比特。每个子空间内应用不变随机旋转以进一步抑制异常值。理论证明该方案为最小化误差的最优混合精度量化策略。在Llama与Qwen2.5系列模型上，我们验证了ResQ优于近期统一及混合精度PTQ方法，在多种基准测试中，相比最优方法SpinQuant，Wikitext上的困惑度降低达33%，推理速度较16比特基线提升最高3倍。代码已公开于https://github.com/utkarsh-dmx/project-resq。
原文摘要 · Abstract (English)
Post-training quantization (PTQ) of large language models (LLMs) holds the promise in reducing the prohibitive computational cost at inference time. Quantization of all weight, activation and key-value (KV) cache tensors to 4-bit without significantly degrading generalizability is challenging, due to the high quantization error caused by extreme outliers in activations. To tackle this problem, we propose ResQ, a PTQ method that pushes further the state-of-the-art. By means of principal component analysis (PCA), it identifies a low-rank subspace (in practice 1/8 of the hidden dimension) in which activation variances are highest, and keep the coefficients within this subspace in high precision, e.g. 8-bit, while quantizing the rest to 4-bit. Within each subspace, invariant random rotation is applied to further suppress outliers. We show that this is a provably optimal mixed precision quantization scheme that minimizes error. With the Llama and Qwen2.5 families of models, we demonstrate that ResQ outperforms recent uniform and mixed precision PTQ methods on a variety of benchmarks, achieving up to 33\% lower perplexity on Wikitext than the next best method SpinQuant, and upto 3\times speedup over 16-bit baseline. Code is available at https://github.com/utkarsh-dmx/project-resq.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。