针对大模型量化精度下降问题，提出感知敏感度的混合精度方法。
MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference
- 按层和列的敏感度动态分配4/6/8位尾数精度，无需训练
- 在W4A4KV4下，LLaMA-2-70B困惑度降至3.77，优于现有方法
- 兼容硬件微缩放格式，适合部署在支持MXINT的设备上
4-bit量化虽能提升大模型推理效率，但因异常值导致显著精度损失。已有方法通过数据旋转或混合精度整数量化缓解，但常依赖软件管理缩放和频繁反量化，带来较大开销。微缩放格式如MXINT通过硬件编码缩放消除这些低效，却与旋转方法不兼容。我们分析发现异常值严重程度各异，量化敏感度在层与列间分布不均，据此提出细粒度、敏感度引导的方案。MXSens为一种免训练方法，基于列与层的敏感度分配4/6/8位尾数宽度，自然利用MXINT的块结构。在多种模型与任务上表现优于当前最佳量化方法。在W4A4KV4设置下，对LLaMA-2-70B和LLaMA-3-8B在WikiText-2上的困惑度分别达3.77和7.63，显著优于基线。本工作为大模型量化建立了精度与资源效率的新平衡。
原文摘要 · Abstract (English)
4-bit quantization enables efficient LLM inference, but suffers from significant accuracy degradation due to outliers. Prior work addresses this problem via data rotation or mixed-precision integer quantization, but often relies on software-managed scaling and frequent dequantization, incurring substantial overhead. Microscaling formats, such as MXINT, eliminate these inefficiencies by encoding scales in hardware, yet remain incompatible with rotation-based methods. Our analysis reveals that outliers vary in severity, from rare extremes to frequent mild deviations, and that quantization sensitivity is unevenly distributed across layers and columns. These insights motivate a fine-grained, sensitivity-guided approach. We introduce MXSens, a training-free method that assigns mixed mantissa bitwidths (4/6/8) based on column- and layer-wise sensitivity, naturally leveraging the block-wise structure of MXINT. MXSens outperforms state-of-the-art quantization methods across a range of models and tasks. Under the W4A4KV4 setting, MXSens achieves perplexities of 3.77 and 7.63 on LLaMA-2-70B and LLaMA-3-8B, respectively, substantially improving over existing baselines on WikiText-2. Our work establishes a new balance between accuracy and resource efficiency for LLM quantization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。