针对专家模型设计分层量化分配策略,显著提升低比特压缩下的语言模型质量。
Q-Strata: Hierarchical Bit Allocation for Mixed-Precision Quantization of Mixture-of-Experts LLMs
- 分两层优化:先局部筛选候选方案,再全局评估模型性能
- 在低比特下比现有方法更低的WikiText2困惑度
- 适合需要高效压缩专家模型的研究者与工程师
混合精度量化(MPQ)为大语言模型的每个线性层分配不同比特数,在固定预算下最小化量化损失。然而,混合专家(MoE)模型在每个专家中都包含这些层,导致分配空间远超密集模型。现有方法要么在每块内统一分配,要么通过可加代理跨块分配,均无法直接优化耦合各块的模型级目标。我们提出Q-Strata,一种双层分配器:内层用廉价代理对每块内的分配进行排序,外层基于组装后的量化模型评估模型级目标,设定每块一个预算而非每个层的比特数。将搜索简化为每块一个预算后,外层可直接优化该目标,捕捉到可加代理忽略的块间耦合。在Mixtral-8x7B-Instruct、Qwen1.5-MoE-A2.7B和DeepSeek-V2-Lite上,Q-Strata在低比特范围内持续取得低于统一比特率GPTQ以及最先进的MoE MPQ方法MxMoE和GEMQ的WikiText2困惑度。代码已公开于https://github.com/snu-mllab/Q-Strata/tree/main。
原文摘要 · Abstract (English)
Mixed-precision quantization (MPQ) assigns a different bitwidth to each linear layer of a large language model (LLM) to minimize the quantization-induced quality loss under a fixed budget, but Mixture-of-Experts (MoE) models contain these layers in every expert of every MoE block, so the allocation space grows far larger than in a dense model. Existing methods either allocate within each block under a uniform per-block budget, or allocate across blocks through an additive proxy, and neither directly optimizes a model-level objective over the choices that couple the blocks. We propose Q-Strata, a bi-level allocator that ranks within-block assignments with a cheap proxy and allocates across blocks with a model-level objective evaluated on the assembled quantized model. Its inner stage caches a Pareto frontier of candidates per block over finely spaced budgets, leaving the outer stage to set one budget per block instead of a bitwidth for every linear layer. With the search reduced to one budget per block, the outer stage optimizes this model-level objective directly, capturing the inter-block coupling that additive proxies miss. On Mixtral-8x7B-Instruct, Qwen1.5-MoE-A2.7B, and DeepSeek-V2-Lite, Q-Strata consistently achieves lower WikiText2 perplexity than uniform-bitwidth GPTQ and the state-of-the-art MoE MPQ methods MxMoE and GEMQ in the low-bit regime. The code is available at https://github.com/snu-mllab/Q-Strata/tree/main.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。