2比特量化模型实现超低功耗推理，性能远超现有方案。
Pushing the Envelope of LLM Inference with Ultra-Low-Bit Quantized Models
- 自底向上设计CPU与GPU的2比特微核，逼近理论峰值性能。
- 2比特推理比16比特快7倍，较当前最优方案提速2.2倍。
- 适配vLLM框架，在Xe2 GPU上实现接近最优的混合精度加速。
超低比特大语言模型（如2比特）在保持接近全精度模型的困惑度与任务准确率的同时，有望显著降低延迟、内存、吞吐量和能耗。然而，其运行时效率尚未充分探索。本文从底层出发，首先在现代CPU上实现2比特微核，达到接近理论峰值性能；集成至推理流水线后，2比特模型相比SOTA的bitnet.cpp运行时提升2.2倍，并实现相较于16比特推理最高7倍的加速。进一步扩展至Intel Xe2 GPU，实现混合精度2比特核，性能接近最优。将这些核集成至vLLM框架，在多种模型与Xe2 GPU上评估端到端推理，最高获得6.7倍于16比特管道的加速，大幅推进了大模型推理的边界。
原文摘要 · Abstract (English)
The advent of ultra-low-bit LLM models, approaching the perplexity and task accuracy of their full precision counterparts, is ushering in a new era of LLM inference. While these advances promise models that are cost-effective regarding latency, memory, throughput, and energy consumption, the efficiency of runtimes for deploying ultra-low-bit models remains under-explored. In this work, we take a bottom-up approach: we first implement 2-bit microkernels for modern CPUs, achieving close-to-roofline performance. We integrate these microkernels into LLM inference pipelines and present end-to-end results with 2-bit models, outperforming the state-of-the-art (SOTA) bitnet.cpp runtime by 2.2$\times$, and deliver up to 7$\times$ speedup compared to 16-bit inference. We extend this work to Intel Xe2 GPUs where we implement mixed-precision, 2-bit kernels, and show their performance to be close-to-optimal. We integrated the GPU kernels in the vLLM framework and evaluated end-to-end inference for a range of models and Xe2 GPUs. We obtain up to 6.7$\times$ speedup compared to the 16-bit pipeline, pushing the envelope of LLM inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。