提出多包络双二值化，提升极端低比特大模型量化性能。
More Than Bits: Multi-Envelope Double Binary Factorization for Extreme Quantization
- 用多包络结构替代单包络，增强精度表达能力。
- 在相同比特下，比现有二值化方法更优，零样本准确率更高。
- 适合部署在资源受限场景的低比特大模型应用。
针对大语言模型的极端低比特量化，双二值化（DBF）因其高效推理且不损失精度而备受关注。然而，DBF的缩放参数过于受限：在分离符号后，所有秩分量共享相同的幅度分布，导致性能饱和。本文提出多包络双二值化（MDBF），保留共享的一对1比特符号基，但将单个包络替换为秩为$l$的包络。通过在包络分量间共享符号矩阵，MDBF有效维持了二值载体，并将有限的存储预算用于提升幅度表达力。同时引入闭式初始化与交替优化方法以优化MDBF。在LLaMA和Qwen系列模型上，MDBF在与先前二值格式相同每权值比特数条件下，提升了困惑度与零样本准确率，且保持相同的部署友好推理原语。
原文摘要 · Abstract (English)
For extreme low-bit quantization of large language models (LLMs), Double Binary Factorization (DBF) is attractive as it enables efficient inference without sacrificing accuracy. However, the scaling parameters of DBF are too restrictive; after factoring out signs, all rank components share the same magnitude profile, resulting in performance saturation. We propose Multi-envelope DBF (MDBF), which retains a shared pair of 1-bit sign bases but replaces the single envelope with a rank-$l$ envelope. By sharing sign matrices among envelope components, MDBF effectively maintains a binary carrier and utilizes the limited memory budget for magnitude expressiveness. We also introduce a closed-form initialization and an alternating refinement method to optimize MDBF. Across the LLaMA and Qwen families, MDBF enhances perplexity and zero-shot accuracy over previous binary formats at matched bits per weight while preserving the same deployment-friendly inference primitive.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。