提出TORQ方法,让4位浮点量化更准,显著提升大模型推理精度。
TORQ: Two-Level Orthogonal Rotation for MXFP4 Quantization

- 通过两级正交旋转重构激活分布几何结构,解决量化误差根源问题。
- 在Qwen3-32B上将困惑度降至8.43,准确率从38.4%提升至73.63%。
- 无需训练的后训练量化方案,适合部署低比特大模型的工程团队。
随着大语言模型迈向实际部署,微缩浮点4(MXFP4)格式因其高动态范围与硬件效率的平衡,成为下一代低比特推理的核心。然而,直接将MXFP4用于激活量化的性能下降严重。本文理论分析了其误差结构,揭示两大失衡:块间方差差异过大与块内码本利用率不均。为此提出无训练后训练量化框架TORQ,通过两级正交旋转重塑激活空间几何特性:宏观层面利用Schur-Horn定理进行块间正交旋转,均衡能量分布;微观层面采用最大熵引导的块内旋转,缓解码本坍塌。在主流模型如LLaMA3和Qwen3上的实验表明,相比现有方法,TORQ显著提升精度——在Qwen3-32B上,WikiText困惑度降至8.43(对比BF16的7.61),平均准确率从直接RTN的38.40%提升至73.63%(接近BF16的74.82%),大幅缩小4比特浮点量化与全精度推理的差距。
原文摘要 · Abstract (English)
As Large Language Models (LLMs) advance toward practical deployment, the Microscaling FP4 (MXFP4) format has emerged as a cornerstone for next-generation low-bit inference, owing to its ability to balance high dynamic range with hardware efficiency. However, directly applying MXFP4 to LLM activation quantization inevitably leads to significant accuracy degradation. In this paper, we theoretically analyze the error structure of MXFP4 activation quantization, revealing that the root cause of this performance drop lies in two structural imbalances between activation distributions and the MXFP4 block floating-point format: (1) extreme inter-block variance imbalance and (2) intra-block codebook utilization imbalance. To address these challenges, we propose TORQ (Two-level Orthogonal Rotation for MXFP4 Quantization), a training-free Post-Training Quantization (PTQ) framework designed to reshape the geometric properties of the activation space through optimal coordinate transformations. At the macroscopic level, TORQ leverages the Schur-Horn theorem to redistribute activation energy via inter-block orthogonal rotation, preventing high-variance blocks from driving up shared scaling factors and thereby preserving the precision of small-magnitude elements. At the microscopic level, TORQ employs maximum-entropy-guided intra-block rotation to alleviate codebook collapse and maximize the MXFP4 codebook's information capacity. Experiments on mainstream LLMs such as LLaMA3 and Qwen3 show that TORQ significantly improves the accuracy of MXFP4 activation quantization compared to existing methods: on Qwen3-32B, the perplexity on WikiText is reduced to 8.43 (vs. 7.61 for BF16), and the average accuracy increases from 38.40% with direct RTN to 73.63% (vs. 74.82% for BF16), substantially narrowing the gap between 4-bit floating-point quantization and full-precision inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。