提出自适应变换WUSH，提升大模型低比特量化精度与速度。
WUSH: Near-Optimal Adaptive Transforms for LLM Quantization
- 设计可学习的块级线性变换，结合哈达玛基底与数据相关成分。
- 在Llama-3.1-8B上，比最优哈达玛基线提升2.8分（RTN）和0.7分（GPTQ）。
- 适合追求高精度低比特推理的开发者，尤其支持FP4加速部署。
量化大模型权重和激活是高效部署的标准方法，但少数极端异常值会拉伸动态范围，放大低比特量化误差。以往基于变换的方法（如哈达玛旋转）固定且数据无关，其量化最优性尚未明确。本文推导出在标准RTN AbsMax缩放块量化器下，针对权重与激活联合量化的闭式最优线性块变换，涵盖整数与浮点格式。所提方法WUSH以哈达玛为骨架，加入数据依赖的二阶矩组件，形成非正交变换，在温和假设下对FP和INT量化器具有近似最优性，并支持高效的融合GPU实现。实验表明，相比最强的哈达玛基线，WUSH在Llama-3.1-8B-Instruct上，于MXFP4中实现+2.8平均准确率提升（使用RTN）和+0.7（使用GPTQ），同时通过FP4 MatMul实现每层最高5.8倍吞吐率超越BF16。源代码见https://github.com/IST-DASLab/WUSH。
原文摘要 · Abstract (English)
Quantizing LLM weights and activations is a standard approach for efficient deployment, but a few extreme outliers can stretch the dynamic range and amplify low-bit quantization errors. Prior transform-based mitigations (e.g., Hadamard rotations) are fixed and data-agnostic, and their optimality for quantization has remained unclear. We derive closed-form optimal linear blockwise transforms for joint weight-activation quantization under standard RTN AbsMax-scaled block quantizers, covering both integer and floating-point formats. The resulting construction, WUSH, combines a Hadamard backbone with a data-dependent second-moment component to form a non-orthogonal transform that is provably near-optimal for FP and INT quantizers under mild assumptions while admitting an efficient fused GPU implementation. Empirically, WUSH improves W4A4 accuracy over the strongest Hadamard-based baselines (e.g., on Llama-3.1-8B-Instruct in MXFP4, it gains +2.8 average points with RTN and +0.7 with GPTQ) while delivering up to 5.8$\times$ per-layer throughput over BF16 via FP4 MatMul. Source code is available at https://github.com/IST-DASLab/WUSH.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。