用多个网格动态选最优4比特表示，提升大模型量化精度。
Grid Games: The Power of Multiple Grids for Quantizing Large Language Models
- 为每组参数选择最优的双网格组合，动态优化量化精度
- 在权重量化和权值+激活联合量化上均显著提升准确率
- 适配TensorCore，适合部署在主流AI硬件上
近期4比特量化（如NVFP4、MXFP4）通过固定浮点网格实现高效压缩。本文提出‘多网格选择’（PO2）机制：对每组数值，可从两个或更多4比特网格中选择更优者，由缩放值中的1位或多比特决定。理论证明，小分组格式（如MXFP、NVFP）在使用PO2时性能显著提升，而大分组下优势消失。实践上，作者设计四种新网格方案：1) PO2(NF4)结合标准网格与学习网格；2) MPO2完全基于真实权重与激活学习；3) PO2(Split87)为带显式零点的非对称网格；4) SFP4是可被TensorCore支持的三重组合（含两个偏移变体）。实验表明，在标准开源模型后训练量化及类Llama模型预训练中，自适应网格在仅权重或权值+激活联合量化下均持续优于单网格4比特格式。代码已开源。
原文摘要 · Abstract (English)
A major recent advance in quantization is given by microscaled 4-bit formats such as NVFP4 and MXFP4, quantizing values into small groups sharing a scale, assuming a fixed floating-point grid. In this paper, we study the following natural extension: assume that, for each group of values, we are free to select the "better" among two or more 4-bit grids marked by one or more bits in the scale value. We formalize the power-of-two-grids (PO2) problem, and provide theoretical results showing that practical small-group formats such as MXFP or NVFP can benefit significantly from PO2 grids, while the advantage vanishes for very large groups. On the practical side, we instantiate several grid families, including 1) PO2(NF4), which pairs the standard NF4 normal grid with a learned grid, 2) MPO2, a grid pair that is fully learned over real weights and activations, 3) PO2(Split87), an explicit-zero asymmetric grid and 4) SFP4, a TensorCore-implementable triple which pairs NVFP4 with two shifted variants. Results for post-training quantization of standard open models and pre-training of Llama-like models show that adaptive grids consistently improve accuracy vs single-grid FP4 under both weight-only and weight+activation. Source code is available at https://github.com/IST-DASLab/GridGames.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。