提出MicroMix，让大模型在黑魔芯片上实现高效混合精度推理。
MicroMix: Efficient Mixed-Precision Quantization with Microscaling Formats for Large Language Models
- 设计微型缩放格式，支持FP4/FP6/FP8混合精度计算。
- 在Llama和Qwen模型上实现接近FP16的准确率，平均精度5比特。
- 适用于需高性能推理的端侧或桌面级显卡用户。
量化通过将高精度矩阵替换为低精度版本，显著加速大语言模型的推理。当前权重-激活量化多聚焦于将两者映射至INT4格式。尽管NVIDIA黑魔架构的新型FP4张量核心相比FP16可实现最高4倍加速，但现有基于INT4的内核因数据格式不匹配，未能充分利用此能力。为此，我们提出MicroMix，一种与Microscaling（MX）数据格式协同设计的混合精度量化算法及GEMM内核。该内核专为黑魔架构优化，支持任意组合的MXFP4、MXFP6和MXFP8通道，并输出BFloat16结果。为在每层线性变换中平衡精度与效率，我们引入量化阈值，识别低精度格式（如MXFP4或MXFP6）会引入过大误差的激活元素，从而选择性分配更高精度通道以保持准确性。在Llama和Qwen模型族上，MicroMix在多种下游任务中实现接近FP16的性能，平均精度达5比特。尤其在Qwen2.5-32B-Base、Coder及Math基准测试中，零样本、代码生成与数学推理任务均实现无损精度。在RTX 5070Ti笔记本与RTX 5090 GPU上，相比TensorRT-FP16，本内核实现2.29-3.38倍加速。代码已开源：https://github.com/lwy2020/MicroMix。
原文摘要 · Abstract (English)
Quantization significantly accelerates inference in large language models (LLMs) by replacing original high-precision matrices with low-precision counterparts. Recent advances in weight-activation quantization have primarily focused on mapping both weights and activations to the INT4 format. Although the new FP4 Tensor Cores in NVIDIA's Blackwell architecture offer up to 4x speedup over FP16, existing INT4-based kernels fail to fully exploit this capability due to mismatched data formats. To bridge this gap, we propose MicroMix, a co-designed mixed-precision quantization algorithm and GEMM kernel based on Microscaling (MX) data formats. Tailored for the Blackwell architecture, the MicroMix kernel supports arbitrary combinations of MXFP4, MXFP6, and MXFP8 channels, and produces BFloat16 outputs. To achieve a favorable trade-off between accuracy and efficiency for each linear layer, we introduce quantization thresholds that identify activation elements where lower-precision formats (MXFP4 or MXFP6) incur excessive quantization error. Our algorithm selectively allocates higher-precision channels to preserve accuracy while maintaining compute efficiency. On the Llama and Qwen model families, MicroMix achieves near-FP16 performance across diverse downstream tasks with an average precision of 5 bits. In particular, Qwen2.5-32B-Base, Coder and Math exhibit lossless accuracy on zero-shot, code generation, and mathematical reasoning benchmarks. In addition, on RTX 5070Ti laptop and RTX 5090 GPUs, our kernel achieves 2.29-3.38x acceleration compared to TensorRT-FP16. Our code is available at https://github.com/lwy2020/MicroMix.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。