arXiv:2512.02901cs.LGcs.AI2025-12被引 2

将已有大模型转为复数形式,实现2比特极低精度推理。

Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in $\{\pm 1, \pm i\}$

  • 把现有实数模型转为复数域的广义线性形式,保持性能不变。
  • 在LLaMA-2 7B上实现2比特量化,性能接近全精度模型。
  • 适合想用低成本硬件部署大模型的研究者和开发者。

大型语言模型(LLMs)虽推动了人工智能发展,但其庞大的内存与计算需求促使量化向单比特极限逼近。复数域模型如iFairy相比实数模型更利于低比特表示,但需从头训练,无法利用现有预训练实数模型。本文提出Fairy2i,一种通用框架,可将预训练实数层转化为等效的广义线性复数形式,实现极低比特量化并复用现有检查点。通过证明实数与广义线性映射的无损数学等价性,将标准Transformer转至复数域,并采用相位感知量化方案,使用四次单位根高效码本。引入递归残差量化机制,迭代最小化量化误差,使推理可通过无乘法累加高效进行。实验表明,Fairy2i在有效2比特精度下恢复了LLaMA-2 7B性能,显著优于现有实数二值化与三值化方法。该工作弥合了复数运算的表征效率与预训练模型实用性之间的差距,为通用硬件上的高效推理开辟新路径。模型与代码已开源:https://huggingface.co/PKU-DS-LAB/Fairy2i-W2 及 https://github.com/PKULab1806/Fairy2i-W2。

原文摘要 · Abstract (English)

Large language models (LLMs) have revolutionized artificial intelligence, yet their massive memory and computational demands necessitate aggressive quantization, increasingly pushing representations toward the theoretical limit of a single bit. While complex-valued LLMs, such as iFairy, offer a superior chance for low-bit representation compared to real-valued counterparts, they require training from scratch, preventing the utilization of the vast ecosystem of pre-trained real-valued foundation models. Here we present Fairy2i, a universal framework that transforms pre-trained real-valued layers into an equivalent widely-linear complex form, enabling extremely low-bit quantization while reusing existing checkpoints. By proving a lossless mathematical equivalence between real and widely-linear maps, we convert standard Transformers into the complex domain and employ a phase-aware quantization scheme with a highly efficient codebook of fourth roots of unity. Furthermore, we introduce a recursive residual quantization mechanism that iteratively minimizes quantization error, allowing inference to proceed via efficient multiplication-free accumulation. We demonstrate that Fairy2i restores the performance of LLaMA-2 7B at an effective 2-bit precision to levels nearly comparable with full-precision baselines, significantly outperforming state-of-the-art real-valued binary and ternary quantization methods. This work bridges the gap between the representational efficiency of complex-valued arithmetic and the practical utility of pre-trained models, paving a new way for efficient inference on commodity hardware. We open-source the Fairy2i model and code at https://huggingface.co/PKU-DS-LAB/Fairy2i-W2 and https://github.com/PKULab1806/Fairy2i-W2.

大模型量化复数神经网络低比特推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。