通过增强残差通道提升NVFP4量化,让大模型推理更快更准。
ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs
- 用量化残差通道增强激活矩阵,保持统一精度格式。
- 在LLaMA和Qwen上达到接近全精度的准确率,速度比FP16快3倍。
- 适合追求高效推理的大模型部署者,尤其看重硬件兼容性。
细粒度数值格式如NVFP4为大语言模型(LLM)高效推理带来了新机遇。然而,现有后训练量化(PTQ)方法难以适配:基于旋转的方法破坏细粒度块隔离;平滑技术在4位量化下误差显著;混合精度方案常与统一精度计算的硬件限制冲突。为此,我们提出ARCQuant框架,通过增强残差通道提升NVFP4性能。不同于牺牲块隔离或硬件统一性的方法,ARCQuant通过在激活矩阵中加入量化残差通道,严格维持统一的NVFP4格式。该设计将误差补偿直接嵌入矩阵压缩维度,可使用标准高效GEMM核,开销极小。理论分析表明,双阶段NVFP4量化的最坏情况误差与MXFP8等标准8位格式相当。在LLaMA和Qwen模型上的大量实验显示,ARCQuant达到当前最优准确率,困惑度和下游任务表现接近全精度基线。在RTX 5090和RTX PRO 6000 GPU上的部署验证了实际效益,速度最高达FP16的3倍。代码已开源:https://github.com/actypedef/ARCQuant。
原文摘要 · Abstract (English)
The emergence of fine-grained numerical formats like NVFP4 presents new opportunities for efficient Large Language Model (LLM) inference. However, it is difficult to adapt existing Post-Training Quantization (PTQ) strategies to these formats: rotation-based methods compromise fine-grained block isolation; smoothing techniques struggle with significant 4-bit quantization errors; and mixed-precision approaches often conflict with hardware constraints on unified-precision computation. To address these challenges, we propose ARCQuant, a framework that boosts NVFP4 performance via Augmented Residual Channels. Distinct from methods that compromise block isolation or hardware uniformity, ARCQuant maintains a strictly unified NVFP4 format by augmenting the activation matrix with quantized residual channels. This design integrates the error compensation process directly into the matrix reduction dimension, enabling the use of standard, highly optimized GEMM kernels with minimal overhead. Theoretical analysis confirms that the worst-case error bound of our dual-stage NVFP4 quantization is comparable to that of standard 8-bit formats such as MXFP8. Extensive experiments on LLaMA and Qwen models demonstrate that ARCQuant achieves state-of-the-art accuracy, comparable to full-precision baselines in perplexity and downstream tasks. Furthermore, deployment on RTX 5090 and RTX PRO 6000 GPUs confirms practical benefits, achieving up to 3x speedup over FP16. Our code is available at https://github.com/actypedef/ARCQuant.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。