无需搜索的细粒度量化框架，让大模型更省内存且推理更快。
SFMP: Fine-Grained, Hardware-Friendly and Search-Free Mixed-Precision Quantization for Large Language Models
- 用连续分数位宽替代离散选择，避免耗时搜索。
- 在保持硬件友好前提下实现逐块细粒度精度分配。
- 支持任意平均位宽的统一计算核，提升推理效率。
混合精度量化是压缩大语言模型以适应严苛内存限制的有力方法。然而，现有方法通常存在两类缺陷：一是依赖昂贵的离散优化来确定精度分配；二是因不规则内存布局导致硬件效率低下。本文提出SFMP，一种无搜索、硬件友好的大语言模型混合精度量化框架。该框架基于四项创新：1）分数位宽，将权重矩阵的整数位宽扩展为连续值，将离散精度分配转化为连续问题；2）块级混合精度，在保持硬件友好性的同时实现权重矩阵内的细粒度精度控制；3）行-列权重重排，通过行列重排聚合显著权重，仅引入少量推理时激活重排开销；4）统一GEMM核，支持任意平均位宽的混合精度矩阵乘法。大量实验表明，SFMP在相同内存约束下优于现有分层式混合精度方法，同时大幅降低量化成本并提升推理效率。代码已开源：https://github.com/Nkniexin/SFMP。
原文摘要 · Abstract (English)
Mixed-precision quantization is a promising approach for compressing large language models under tight memory budgets. However, existing mixed-precision methods typically suffer from one of two limitations: they either rely on expensive discrete optimization to determine precision allocation, or introduce hardware inefficiencies due to irregular memory layouts. We propose SFMP, a search-free and hardware-friendly mixed-precision quantization framework for large language models. The framework is built upon four novel ideas: Fractional bit-width, which extends integer bit-width for weight matrix to fractional value and transforms discrete precision allocation as a continuous problem; 2)Block-wise mixed-precision, enabling fine-grained precision within weight matrices while remaining hardware-friendly; 3)Row-column weight reordering, which aggregates salient weights via row and column reordering, incurring only a small activation reordering overhead during inference; 4)Unified GEMM kernel, which supports mixed-precision GEMM at arbitrary average bit-width. Extensive experiments demonstrate that SFMP outperforms state-of-the-art layer-wise mixed-precision methods under the same memory constraints, while significantly reducing quantization cost and improving inference efficiency. Code is available at https://github.com/Nkniexin/SFMP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。