SignRoundV2让大模型在2比特压缩下仍保持高精度，突破低比特量化性能瓶颈。
SignRoundV2: Toward Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMs
- 基于梯度与重建误差自适应分配每层比特数，实现混合精度量化。
- 在4.5比特平均下仅损失约1%性能，2比特权重量化显著提效。
- 适合追求极致压缩的模型部署场景，尤其适用于资源受限设备。
极低比特量化对高效部署大语言模型至关重要，但通常在2比特甚至4比特（如MXFP4）时导致严重性能下降。我们提出SignRoundV2，一种后训练量化框架，在极端压缩下仍能保持高性能。该方法引入（1）一种简单高效的自适应混合精度策略，利用梯度信息和量化引起的重建误差指导逐层比特分配；（2）一系列轻量级稳定技术，包括损失过滤和预调优缩放搜索，以提升极低比特环境下的调优效果。实验表明，SignRoundV2在多种大模型上实现近无损性能，在混合MXFP设置下平均4.5比特时性能差距缩小至约1%，并在极具挑战性的2比特权重量化中显著提升准确率。代码已开源。
原文摘要 · Abstract (English)
Extremely low-bit quantization is critical for efficiently deploying Large Language Models (LLMs), yet it often leads to severe performance degradation at 2 bits and even at 4 bits (e.g., MXFP4). We present SignRoundV2, a post-training quantization framework designed to maintain high performance even under aggressive compression. SignRoundV2 introduces (1) a simple yet efficient adaptive mixed-precision strategy that leverages gradient information and quantization-induced reconstruction errors to guide layer-wise bit allocation, and (2) a set of lightweight stabilization techniques, including loss filtering and a pre-tuning scale search, to improve tuning effectiveness in extremely low-bit regimes. Our approach takes a significant step toward closing the performance gap between quantized and full-precision models. Experimental results across diverse LLMs demonstrate that SignRoundV2 achieves near-lossless performance in mixed MXFP settings, narrowing the gap to $\sim$1\% at an average of 4.5 bits, while substantially improving accuracy in challenging 2-bit weight-only quantization. The source code is available at \url{https://github.com/intel/auto-round}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。