3比特量化提升大模型精度，通过旋转降噪实现接近浮点的推理效果。
ITQ3_S: High-Fidelity 3-bit LLM Inference via Interleaved Ternary Quantization with Rotation-Domain Smoothing
- 用快速沃尔什-哈达玛变换预旋转权重，使分布更接近高斯分布
- 重建误差严格小于普通3比特方法，实测在RTX 5090上超越4比特性能
- 专为消费级显卡设计，融合反变换到CUDA核中，无额外误差
我们提出ITQ3_S（交织三值量化——专用版），一种新型3比特大模型权重量化格式，结合基于快速沃尔什-哈达玛变换（FWHT）的旋转域策略TurboQuant（TQ）。传统3比特方法因权重分布尾部过重和通道间异常值导致精度下降。ITQ3_S在量化前通过FWHT对权重空间进行预旋转，将异常能量分散至向量各处，形成近似高斯分布，适合均匀三值编码。我们推导出严格的反量化过程，将256点逆FWHT融入CUDA共享内存加载阶段，确保重构误差仅由三值量化网格决定，不引入变换逆带来的额外误差。对于任意权重向量$\w] ∈ ℝ^{256}$，重构满足$\|\|\hat{\mathbf{w}} - \mathbf{w}\|\|_2 ≤ ε_q$，严格优于未利用旋转归一化的均匀3比特基线。TurboQuant无原生CUDA内核，直接集成会引入域不匹配误差，逐层累积导致质量低于标准3比特。ITQ3_S通过协同设计FWHT旋转与量化核，构建统一流水线，基于IQ3_S权重格式，并将逆变换融合进CUDA MMQ核。实验表明，在NVIDIA RTX 5090（Blackwell）上，ITQ3_S在困惑度上媲美FP16，同时吞吐量超过4比特方案1.5倍以上，得益于优化的DP4A与张量核心调度。结果确立了ITQ3_S作为消费级硬件上高保真大模型部署的实用且数学严谨的解决方案。
原文摘要 · Abstract (English)
We present ITQ3_S (Interleaved Ternary Quantization -- Specialized), a novel 3-bit weight quantization format for LLMs integrating TurboQuant (TQ), a rotation-domain strategy based on the Fast Walsh-Hadamard Transform (FWHT). Conventional 3-bit methods suffer precision loss from heavy-tailed weight distributions and inter-channel outliers. ITQ3_S pre-rotates the weight space via FWHT before quantization, spreading outlier energy across the vector and inducing a near-Gaussian distribution amenable to uniform ternary coding. We derive a rigorous dequantization procedure fusing a 256-point Inverse FWHT into the CUDA shared-memory loading stage, ensuring reconstruction error is bounded exclusively by the ternary quantization grid with no additional error from the transform inversion. For any weight vector $\mathbf{w} \in \mathbb{R}^{256}$, the reconstruction satisfies $\|\hat{\mathbf{w}} - \mathbf{w}\|_2 \leq ε_q$, strictly smaller than uniform 3-bit baselines that do not exploit rotation-induced distribution normalization. TurboQuant lacks a native CUDA kernel, precluding direct deployment; naively composing TQ with existing weight quantizers introduces domain mismatch errors that accumulate across layers, degrading quality below standard 3-bit baselines. ITQ3_S resolves this by co-designing the FWHT rotation and quantization kernel as a unified pipeline grounded in the IQ3_S weight format, with the inverse transform fused into the CUDA MMQ kernel. Empirically, on the NVIDIA RTX 5090 (Blackwell), ITQ3_S achieves perplexity competitive with FP16 while delivering throughput exceeding 1.5x that of 4-bit alternatives via optimized DP4A and Tensor Core scheduling. Our results establish ITQ3_S as a practical, mathematically grounded solution for high-fidelity LLM deployment on consumer hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。