用奇异值分解解决大模型量化中的激活异常，实现4比特高效推理。
QUAD: Quantization and Parameter-Efficient Tuning of LLM with Activation Decomposition
- 通过奇异值分解将激活异常转移至高精度维度，其余部分压缩至4比特。
- 在Llama-3和Qwen-2.5上实现94%~96%准确率，启用微调后达98%。
- 支持参数高效微调，适合部署资源受限的大模型场景。
大型语言模型（LLMs）在多种应用中表现优异，但因规模庞大导致效率低下。尽管量化可降低计算成本，现有方法在中等规模模型（如 Llama-3-8B）上因激活异常而损失精度。为此，我们提出 QUAD（基于激活分解的量化），利用奇异值分解（SVD）抑制激活异常，实现有效的4比特量化。QUAD 使用校准数据离线估计激活奇异向量，构建正交变换矩阵 P，将异常值映射到全精度附加维度，其余成分量化为4比特。此外，QUAD 通过可调全精度异常权重实现参数高效微调，缩小了量化与全精度模型间的准确率差距。实验表明，对于 Llama-3 与 Qwen-2.5 模型，在 W4A4 量化下准确率达 94% ~ 96%，结合 W4A4/A8 与参数高效微调后可达 98%。代码已开源于仓库。
原文摘要 · Abstract (English)
Large Language Models (LLMs) excel in diverse applications but suffer inefficiency due to massive scale. While quantization reduces computational costs, existing methods degrade accuracy in medium-sized LLMs (e.g., Llama-3-8B) due to activation outliers. To address this, we propose QUAD (Quantization with Activation Decomposition), a framework leveraging Singular Value Decomposition (SVD) to suppress activation outliers for effective 4-bit quantization. QUAD estimates activation singular vectors offline using calibration data to construct an orthogonal transformation matrix P, shifting outliers to additional dimensions in full precision while quantizing rest components to 4-bit. Additionally, QUAD enables parameter-efficient fine-tuning via adaptable full-precision outlier weights, narrowing the accuracy gap between quantized and full-precision models. Experiments demonstrate that QUAD achieves 94% ~ 96% accuracy under W4A4 quantization and 98% accuracy with W4A4/A8 and parameter-efficient fine-tuning for Llama-3 and Qwen-2.5 models. Our code is available at \href{https://github.com/hyx1999/Quad}{repository}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。