通过搜索优化块浮点缩放因子,显著降低量化误差。
Search Your Block Floating Point Scales!

- 用细粒度搜索法基于尾数位选择最优缩放因子
- NVFP4量化误差降27%,语言模型精度提15点
- 适合需要高精度低精度推理的生成模型部署
量化已成为加速生成模型推理的标准技术,可实现更快的低精度计算和更少的内存传输。近期GPU加速器已原生支持微缩放块浮点(BFP)格式。现有标准BFP算法采用基于块内最大绝对值的固定缩放因子,但该策略可能并非最优。本文提出ScaleSearch:利用微缩放格式中的尾数位进行细粒度搜索,以最小化给定分布下的量化误差。ScaleSearch可与后训练量化(PTQ)和低精度注意力等方法集成,显著提升性能。我们进一步提出ScaleSearchAttention,一种基于NVFP4的加速注意力算法,结合ScaleSearch与改进技术,在因果语言建模中实现近零性能损失。实验表明,ScaleSearch使NVFP4量化误差降低27%,在MATH500数据集上将Qwen3-8B的PTQ精度提升最高达15点;ScaleSearchAttention使Llama 3.1 70B在Wikitext-2上的困惑度降低最多0.77点。所提方法在接近基线性能的同时显著提升量化精度。
原文摘要 · Abstract (English)
Quantization has emerged as a standard technique for accelerating inference for generative models by enabling faster low-precision computations and reduced memory transfers. Recently, GPU accelerators have added first-class support for microscaling Block Floating Point (BFP) formats. Standard BFP algorithms use a fixed scale based on the maximum magnitude of the block. We observe that this scale choice can be suboptimal with respect to quantization errors. In this work, we propose ScaleSearch, an alternative strategy for selecting these scale factors: using a fine-grained search leveraging the mantissa bits in microscaling formats to minimize the quantization error for the given distribution. ScaleSearch can be integrated with existing quantization methods such as Post Training Quantization and low-precision attention, and is shown to improve their performance. Additionally, we introduce ScaleSearchAttention, an accelerated NVFP4-based attention algorithm, which uses ScaleSearch and adapted prior techniques to ensure near-0 performance loss for causal language modeling. Experiments show that ScaleSearch reduces quantization error by 27% for NVFP4 and improves language model PTQ by up to 15 points for MATH500 (Qwen3-8B), while ScaleSearchAttention improves Wikitext-2 PPL by upto 0.77 points for Llama 3.1 70B. The proposed methods closely match baseline performance while providing quantization accuracy improvements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。