KronQ通过梯度协方差优化量化，显著提升大模型压缩效果。
KronQ: LLM Quantization via Kronecker-Factored Hessian
- 引入梯度协方差构建量化损失，打破输出通道等效假设。
- 2比特量化下在LLaMA-3-70B上实现7.93困惑度，远超对比方法。
- 适合追求高精度低比特量化的研究者与工业部署场景。
后训练量化（PTQ）是无需微调即可压缩大语言模型的常用技术。现有二阶PTQ方法（如GPTQ）基于输入激活统计构建量化目标，隐含假设所有输出通道对层内重构贡献相等。本文提出KronQ，通过引入梯度协方差挑战该假设，在克罗内克分解海森矩阵近似下，量化损失同时依赖激活与梯度协方差，并在两个层面加以利用：(1) 提出双向不一致性处理，将原有输入侧随机旋转扩展至输出维度，利用梯度协方差降低输入与输出方向的权重幅度方差；(2) 推导新的跨层混合精度分配敏感度指标，由梯度与激活海森迹驱动。值得注意的是，在LLaMA-3-70B的2比特权重仅量化场景下，GPTQ与GPTAQ出现发散或退化量化（WikiText-2上困惑度>2000），而KronQ达到7.93的困惑度。
原文摘要 · Abstract (English)
Post-training quantization (PTQ) is a widely adopted technique for compressing large language models (LLMs) without retraining. Most existing second-order PTQ methods, including GPTQ, construct quantization objectives from input activation statistics, effectively assuming that all output channels contribute equally to the layer-wise reconstruction objective. We propose KronQ, a PTQ framework that challenges this assumption by introducing the gradient covariance into the quantization pipeline. Under the Kronecker-factored Hessian approximation, the quantization loss depends jointly on both the activation and gradient covariances, and KronQ exploits this at two complementary levels. (1) KronQ introduces bidirectional incoherence processing, extending the existing input-side random rotation to the output dimension using the gradient covariance, reducing weight magnitude variance across both input and output dimensions. (2) KronQ derives a new sensitivity metric for inter-layer mixed-precision allocation, driven by the gradient and activation Hessian traces. Notably, in the case of 2-bit weight-only quantization on LLaMA-3-70B, while GPTQ and GPTAQ diverge or produce degenerate quantizations (>2000 perplexity on WikiText-2), \KronQ{} achieves 7.93 perplexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。