arXiv:2603.11021cs.LG2026-03被引 2

用24维李奇格网实现高效大模型量化,突破传统压缩瓶颈。

Leech Lattice Vector Quantization for Efficient LLM Compression

  • 基于李奇格网的向量量化,无需存储码本即可高效编码参数
  • 在24维空间实现最优球体密铺,压缩性能超越现有方法
  • 适合追求高精度低延迟推理的大模型部署场景

大语言模型的标量量化受信息论极限制约。向量量化通过联合编码参数块可突破此限制,但实际应用需避免昂贵的查找机制或显式码本存储。格结构方法通过高度有序且密集的排列解决该问题。本文研究24维李奇格格网——目前已知唯一具有最优球体堆积与接触配置的高维格结构。为使李奇格格网适用于大模型量化,我们扩展了基于扩展戈莱码构造的搜索算法:一、支持索引,实现比特串与格点间的无码本转换;二、支持对李奇格格网壳层并集的角域搜索;三、提出完全可并行的反量化核。此外,我们对形状-增益量化与GPTQ风格海森修正的结合进行了几何重解释:标准缩放校正步骤相当于将参数投影到乘积球面上,形成主要作用于方向的球面GPTQ。实验表明,低角畸变的李奇格向量量化显著降低对哈达玛/旋转预处理的敏感性,实现强效的无哈达玛微调。该方法在多个基准上达到当前最佳性能,优于Quip#、QTIP和PVQ等近期方案,验证了高维格结构在可扩展、理论驱动模型压缩中的有效性。

原文摘要 · Abstract (English)

Scalar quantization of large language models (LLMs) is fundamentally limited by information-theoretic bounds. While vector quantization (VQ) overcomes these limits by encoding blocks of parameters jointly, practical implementations must avoid the need for expensive lookup mechanisms or other explicit codebook storage. Lattice approaches address this through highly structured and dense packing. This paper explores the Leech lattice, which, with its optimal sphere packing and kissing configurations at 24 dimensions, is the highest dimensional lattice known with such optimal properties. To make the Leech lattice usable for LLM quantization, we extend an existing search algorithm based on the extended Golay code construction, to i) support indexing, enabling conversion to and from bitstrings without materializing the codebook, ii) allow angular search over union of Leech lattice shells, iii) propose fully-parallelisable dequantization kernel. Lastly, we provide a geometric reinterpretation of combining shape--gain quantization with GPTQ-style Hessian corrections: the standard scale-correction step of shape--gain acts as a retraction onto a product of spheres, yielding a Spherical GPTQ primarily acting on directions. We find that low-angular-distortion LLVQ reduces sensitivity to Hadamard/rotation preprocessing, and enables a strong Hadamard-free PTQ in practice. LLVQ delivers state-of-the-art LLM quantization performance, outperforming recent methods such as Quip\#, QTIP, and PVQ. The results highlight the effectiveness of high-dimensional lattices for scalable, theoretically grounded model compression.

大模型压缩向量量化格量化高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。