arXiv:2507.18553cs.LGcs.DS2025-07被引 21

GPTQ量化方法本质是最近平面算法,为模型压缩提供新理论视角。

The Geometry of LLM Quantization: GPTQ as Babai's Nearest Plane Algorithm

  • 将GPTQ反向执行,等价于求解格上的最近向量问题
  • 避免权重截断后性能超越原GPTQ,误差有理论上限
  • 适合关注量化理论与高性能推理优化的研究者

将大语言模型(LLM)权重从16位量化到更低精度是部署大规模Transformer模型至低成本加速器的通用方法。尽管GPTQ已成为主流的一次性后训练量化方法,但其内部机制被描述为一系列代数更新,缺乏几何解释或最坏情况保证。本文证明:当对线性层从最后一维到第一维反向执行时,GPTQ在数学上等价于基于输入特征的海森矩阵定义的格上的经典最近向量问题(CVP)的Babai最近平面算法。这一等价关系基于复杂数学推导,带来两个分析结果:一是GPTQ的误差传播步骤获得直观几何解释;二是若不进行权重截断,GPTQ继承了Babai算法的误差上界。基于此上界,我们设计出无截断的后训练量化方法,性能优于原始GPTQ。此外,还实现了高效的GPU推理内核。这些成果为GPTQ奠定了坚实的理论基础,并开启引入数十年格算法进展用于未来百亿参数模型量化的新路径。源码见https://github.com/IST-DASLab/GPTQ-Babai。

原文摘要 · Abstract (English)

Quantizing the weights of large language models (LLMs) from 16-bit to lower bitwidth is the de facto approach to deploy massive transformers onto more affordable accelerators. While GPTQ emerged as one of the standard methods for one-shot post-training quantization at LLM scale, its inner workings are described as a sequence of algebraic updates that obscure geometric meaning or worst-case guarantees. In this work, we show that, when executed back-to-front (from the last to first dimension) for a linear layer, GPTQ is mathematically identical to Babai's nearest plane algorithm for the classical closest vector problem (CVP) on a lattice defined by the Hessian matrix of the layer's inputs. This equivalence is based on a sophisticated mathematical argument, and has two analytical consequences: first, the GPTQ error propagation step gains an intuitive geometric interpretation; second, GPTQ inherits the error upper bound of Babai's algorithm under the assumption that no weights are clipped. Leveraging this bound, we design post-training quantization methods that avoid clipping, and outperform the original GPTQ. In addition, we provide efficient GPU inference kernels for the resulting representation. Taken together, these results place GPTQ on a firm theoretical footing and open the door to importing decades of progress in lattice algorithms towards the design of future quantization algorithms for billion-parameter models. Source code is available at https://github.com/IST-DASLab/GPTQ-Babai.

量化格理论大模型推理算法优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。