提出可学习仿射变换,提升大模型微缩量化精度。
LATMiX: Learnable Affine Transformations for Microscaling Quantization of LLMs
- 用可学习仿射变换替代传统旋转,优化激活分布。
- 在多种模型规模上,低比特微缩量化平均准确率显著提升。
- 适合关注大模型高效部署与量化性能的工程师和研究者。
后训练量化(PTQ)是降低大语言模型内存与计算成本的常用方法。近期研究发现,对激活值应用可逆变换能有效提升量化鲁棒性,减少激活异常值;但现有方法多局限于旋转或Hadamard型变换。此外,多数研究聚焦传统量化方案,而现代硬件日益支持微缩量化(MX)格式。此前尝试结合两者时出现严重性能下降,导致先前工作需引入额外假设。本文从互补视角出发,首先通过理论分析推导出在MX量化下量化误差的上界,强调需同时考虑激活分布与量化结构。基于此,我们提出LATMiX,一种使用标准深度学习工具优化的可学习可逆仿射变换方法,实现更优的异常值抑制。实验表明,在多个模型规模及广泛零样本基准测试中,该方法在低比特MX量化下均优于强基线,平均准确率持续提升。
原文摘要 · Abstract (English)
Post-training quantization (PTQ) is a widely used approach for reducing the memory and compute costs of large language models (LLMs). Recent studies have shown that applying invertible transformations to activations can significantly improve quantization robustness by reducing activation outliers; however, existing approaches are largely restricted to rotation or Hadamard-based transformations. Moreover, most studies focused primarily on traditional quantization schemes, whereas modern hardware increasingly supports the microscaling (MX) data format. Attempts to combine both showed severe performance degradation, leading prior work to introduce assumptions on the transformations. In this work, we take a complementary perspective. First, we provide a theoretical analysis of transformations under MX quantization by deriving a bound on the quantization error. Our analysis emphasizes the importance of accounting for both the activation distribution and the underlying quantization structure. Building on this analysis, we propose LATMiX, a method that generalizes outlier reduction to learnable invertible affine transformations optimized using standard deep learning tools. Experiments show consistent improvements in average accuracy for MX low-bit quantization over strong baselines on a wide range of zero-shot benchmarks, across multiple model sizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。