动态调整模型精度，让大模型推理更快更省内存
FlexQuant: A Flexible and Efficient Dynamic Precision Switching Framework for LLM Quantization
- 根据上下文动态调整每层权重的量化位数
- 在多个任务上实现1.3倍加速，精度几乎不变
- 适合需要高效部署大模型的工程师和研究者
大型语言模型的快速发展加剧了因参数规模扩大与硬件能力滞后之间的差距带来的内存瓶颈。尽管后训练量化技术能有效降低内存开销，但现有方法多依赖静态量化策略，难以适应动态工作负载。为此，我们提出FlexQuant，一种动态精度切换框架，旨在优化推理速度与准确率之间的权衡。该框架利用模型困惑度熵与KL散度，实现细粒度、分层的混合精度量化，并在每次生成令牌时动态调整位宽。FlexQuant对量化策略进行了全面分析，提出了最优切换的精度需求模型，并实现了高效的细粒度精度管理。评估结果表明，该框架在多种语言任务中实现了1.3倍的端到端加速，且引入的精度损失可忽略不计。该框架为大模型的高效部署提供了灵活自适应的解决方案。代码已公开于https://github.com/ZongwuWang/FlexQuant.git。
原文摘要 · Abstract (English)
The rapid advancement of large language models (LLMs) has exacerbated the memory bottleneck due to the widening gap between model parameter scaling and hardware capabilities. While post-training quantization techniques effectively reduce memory overhead, existing methods predominantly rely on static quantization strategies, which struggle to adapt to dynamic workloads. To address this, we propose FlexQuant, a dynamic precision-switching framework that optimizes the trade-off between inference speed and accuracy. Leveraging model perplexity entropy and Kullback-Leibler divergence, FlexQuant enables fine-grained, layer-wise mixed-precision quantization and dynamically adjusts bit-widths during each token generation. FlexQuant provides a comprehensive analysis of quantization strategies, introduces a precision requirement model for optimal switching, and implements efficient fine-grained precision management. Evaluations demonstrate that FlexQuant achieves a 1.3x end-to-end speedup across diverse language tasks with negligible accuracy loss introduced. This framework offers a flexible and adaptive solution for efficient LLM deployment. Code is released at https://github.com/ZongwuWang/FlexQuant.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。