QuantX实现3比特量化，保持模型性能且适配硬件高效推理。
QuantX: A Framework for Hardware-Aware Quantization of Generative AI Workloads
- 针对不同硬件设计量化策略，优化推理速度与内存占用。
- 3比特量化后性能仅比原模型低6%，优于当前主流技术。
- 适合需要轻量化部署的生成式AI应用开发者使用。
我们提出QuantX：一套专为大语言模型（LLM）和视觉语言模型（VLM）设计的量化方案。该框架可将模型量化至3比特，同时保持极小性能损失。QuantX的量化策略充分考虑硬件特性，确保推理时高效解量化，实现运行时速度、内存需求与模型精度间的灵活权衡。实验表明，对LlaVa-v1.6进行3比特量化后，多项终端任务性能仅比未量化模型低6%，并超越近期公布的最先进量化方法。我们将QuantX中的某一技术集成至流行的Llama.cpp框架，验证其在实际运行时的可行性，相较主流量化技术表现更优。本文还深入分析了大语言模型量化过程，为QuantX中多样化的量化配方与选项提供了设计依据。
原文摘要 · Abstract (English)
We present QuantX: a tailored suite of recipes for LLM and VLM quantization. It is capable of quantizing down to 3-bit resolutions with minimal loss in performance. The quantization strategies in QuantX take into account hardware-specific constraints to achieve efficient dequantization during inference ensuring flexible trade-off between runtime speed, memory requirement and model accuracy. Our results demonstrate that QuantX achieves performance within 6% of the unquantized model for LlaVa-v1.6 quantized down to 3-bits for multiple end user tasks and outperforms recently published state-of-the-art quantization techniques. We further integrate one particular technique from QuantX into the popular Llama.cpp framework and show its feasibility in terms of runtime compared to the mainstream quantization techniques from Llama.cpp. Lastly, this manuscript provides insights into the LLM quantization process that motivated the range of recipes and options that are incorporated in QuantX.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。