通过重排激活值分布,显著提升量化精度,尤其在小块尺寸下表现突出。
Pushing the Limits of Block Rotations in Post-Training Quantization
- 先重排后旋转,利用贪心算法均衡各块的ℓ1范数以抑制异常值。
- 在16块大小下,量化Llama3 1B至INT4时,困惑度恢复达90%(无重排仅46%)。
- 方法可嵌入模型权重,部署时无额外计算开销,适合大模型轻量化应用。
近期后训练量化(PTQ)方法采用块旋转来分散量化前的异常值。尽管这减少了在线全向量旋转的开销,但块结构对异常值抑制的影响仍不明确。为此,本文首次对块哈达玛旋转的异常值抑制进行系统性、非渐近分析。结果表明,异常值抑制受输入向量几何结构的根本限制;在确定性最坏情况下,当旋转前ℓ1范数质量在各块间均匀分布时,旋转后异常值最少。基于此,我们提出PeRQ(Permute, Rotate, then Quantize)框架,通过预旋转重排激活值质量。设计贪心质量扩散算法,使各块期望ℓ1范数趋于一致。为避免推理开销,识别变压器架构中置换等变区域,将重排合并至模型权重中。实验表明,PeRQ在所有块大小下均提升精度,在块大小为16时,将Llama3 1B量化至INT4,困惑度恢复达90%(无重排仅为46%)。
原文摘要 · Abstract (English)
Recent post-training quantization (PTQ) methods have adopted block rotations to diffuse outliers prior to rounding. While this reduces the overhead of online full-vector rotations, the effect of block structure on outlier suppression remains poorly understood. To fill this gap, we present the first systematic, non-asymptotic analysis of outlier suppression for block Hadamard rotations. Our analysis reveals that outlier suppression is fundamentally limited by the geometry of the input vector. In particular, in the deterministic worst case, post-rotation outliers are minimized when the pre-rotation $\ell_1$ norm mass is evenly distributed across blocks. Guided by these insights, we introduce PeRQ (Permute, Rotate, then Quantize), a PTQ framework that redistributes activation mass via permutations prior to rotation. We propose a greedy mass diffusion algorithm to calibrate permutations by equalizing the expected blockwise $\ell_1$ norms. To avoid adding inference overhead, we identify permutation-equivariant regions in transformer architectures to merge these permutations into model weights before deployment. Experiments show that PeRQ consistently improves accuracy across all block sizes, recovering up to 90% of the full-vector rotation perplexity when quantizing Llama3 1B to INT4 with block size 16, compared to 46% without permutations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。