arXiv:2508.05571cs.LGcs.CL2025-08被引 7

用复数权重突破2比特量化瓶颈,实现高效高精度大模型。

iFairy: the First 2-bit Complex LLM with All Parameters in $\{\pm1, \pm i\}$

  • 将权重映射到±1、±i的四元单位根,实现对称最优2比特表示。
  • 在语言模型上超越现有2比特方法的性能天花板,同时保持极低存储与计算开销。
  • 适合追求极致压缩与推理效率的低比特大模型研究者。

量化感知训练(QAT)将量化融入训练过程,使大语言模型学习鲁棒的低比特表示,被广泛认为是最有前景的研究方向之一。现有所有QAT研究均聚焦于最小化全精度模型的量化误差,以全精度模型的准确率为上限(性能天花板)。目前尚无方法尝试突破此天花板。为打破这一限制,我们提出新范式:先提升天花板(全精度模型),再高效地将其量化至2比特。我们提出Fairy±i,首个面向复数权重的大语言模型2比特量化框架。该方法利用复数域的表示优势提升全精度模型精度,将权重映射至四元单位根{±1, ±i},形成完美对称且信息论最优的2比特表示。重要的是,每个量化权重的实部或虚部必为零,支持仅通过加法和元素交换完成乘法无关推理。实验表明,Fairy±i在困惑度(PPL)与下游任务上均超越现有2比特量化方法的性能上限,同时严格保持存储与计算效率。本工作开辟了极端低比特约束下构建高精度、实用大模型的新路径。

原文摘要 · Abstract (English)

Quantization-Aware Training (QAT) integrates quantization into the training loop, enabling LLMs to learn robust low-bit representations, and is widely recognized as one of the most promising research directions. All current QAT research focuses on minimizing quantization error on full-precision models, where the full-precision accuracy acts as an upper bound (accuracy ceiling). No existing method has even attempted to surpass this ceiling. To break this ceiling, we propose a new paradigm: raising the ceiling (full-precision model), and then still quantizing it efficiently into 2 bits. We propose Fairy$\pm i$, the first 2-bit quantization framework for complex-valued LLMs. Specifically, our method leverages the representational advantages of the complex domain to boost full-precision accuracy. We map weights to the fourth roots of unity $\{\pm1, \pm i\}$, forming a perfectly symmetric and information-theoretically optimal 2-bit representation. Importantly, each quantized weight has either a zero real or imaginary part, enabling multiplication-free inference using only additions and element swaps. Experimental results show that Fairy$\pm i$ outperforms the ceiling of existing 2-bit quantization approaches in terms of both PPL and downstream tasks, while maintaining strict storage and compute efficiency. This work opens a new direction for building highly accurate and practical LLMs under extremely low-bit constraints.

低比特量化复数模型高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。