arXiv:2602.20191cs.LGcs.AI2026-02

提出可动态调整精度的混合位量化方法,让大模型运行更灵活高效。

MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM

  • 基于令牌敏感度动态分配不同精度,用递归残差重建高精度权重。
  • 在任意精度切换下保持性能,内存节省超34%,吞吐提升1.34倍。
  • 适合资源受限场景下的大模型部署,尤其适合动态变化的计算环境。

动态推理延迟和内存限制要求大语言模型(LLM)具备灵活部署能力,可在运行时根据可用算力选择不同量化精度。现有任意精度量化方法或依赖硬件低效的向量量化,或在切换位宽时引入额外缩放因子。同时,针对固定低精度校准的后训练量化(PTQ)方法在运行时精度变动下泛化能力差。本文将此问题归因于精度相关的异常值迁移现象:敏感令牌的分布随精度变化而改变。为此,我们提出 exttt{MoBiQuant},一种新型的任意精度混合位量化框架,通过令牌感知路由机制动态选择每个令牌的最优推理精度,并采用多合一递归残差量化,在运行时迭代重构更高精度权重,有效缓解异常值迁移。大量实验表明, exttt{MoBiQuant} 在单精度量化中达到或超越前沿水平,且具备强弹性,在内存节省和吞吐方面相比最先进任意精度方法最高提升1.34倍。

原文摘要 · Abstract (English)

Dynamic runtime latency and memory constraints necessitate flexible large language model (LLM) deployment, where an LLM can be inferred with various quantization precisions based on available computational resources. Recent work on such any-precision quantization either relies on hardware-inefficient vector quantization or induces additional scaling factors when switching between bit-widths. Meanwhile, existing post-training quantization (PTQ) methods calibrated for a fixed low precision show poor generalizability under runtime precision change. In this work, we attribute the source of poor generalization across bit-widths to a precision-dependent \textit{outlier migration} phenomenon where the distribution of PTQ-sensitive tokens changes across precisions. Motivated by this observation, we propose \texttt{MoBiQuant}, a novel any-precision Mixture-of-Bits quantization framework that adjusts weight precision for flexible LLM inference based on token sensitivity. Specifically, we propose a many-in-one recursive residual quantization that can iteratively reconstruct higher-precision weights at runtime and mitigates \textit{outlier migration} with a token-aware router to dynamically select the optimal inference precision of each token.Extensive experiments show that \texttt{MoBiQuant} matches or surpasses frontier single-precision PTQ while exhibiting strong elasticity, achieving significant memory savings and throughput gains of up to $1.34\times$ over state-of-the-art any-precision methods.

量化大模型任意精度动态调度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。