实测三种量化方式在大模型上的表现，给出部署选型建议。
"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization
- 系统评估FP8、INT8、INT4在全系列Llama-3上的效果。
- FP8几乎无损，INT8仅降1-3%，INT4竟接近8比特性能。
- 根据场景推荐：同步用W4A16，异步用W8A8，混合看具体需求。
量化是加速大语言模型推理的强大工具，但不同格式在精度与性能间的权衡仍不清晰。本文开展迄今最全面的实证研究，评估了FP8、INT8和INT4在完整Llama-3.1模型家族上的表现，覆盖学术基准与真实任务，共完成超过50万次评估。结果表明：(1) FP8（W8A8-FP）在所有模型规模下基本无损失；(2) 经优化的INT8（W8A8-INT）仅造成1%-3%的精度下降；(3) 仅权重为INT4（W4A16-INT）的量化比预期更具竞争力，可媲美8比特量化。进一步通过vLLM框架分析推理性能，提出明确部署建议：在同步场景中W4A16最具成本效益，异步连续批处理则以W8A8为主导。对于混合负载，最优选择取决于具体应用场景。研究提供数据驱动的规模化部署指南，实现速度、效率与精度的最佳平衡。
原文摘要 · Abstract (English)
Quantization is a powerful tool for accelerating large language model (LLM) inference, but the accuracy-performance trade-offs across different formats remain unclear. In this paper, we conduct the most comprehensive empirical study to date, evaluating FP8, INT8, and INT4 quantization across academic benchmarks and real-world tasks on the entire Llama-3.1 model family. Through over 500,000 evaluations, our investigation yields several key findings: (1) FP8 (W8A8-FP) is effectively lossless across all model scales, (2) well-tuned INT8 (W8A8-INT) achieves surprisingly low (1-3\%) accuracy degradation, and (3) INT4 weight-only (W4A16-INT) is more competitive than expected, rivaling 8-bit quantization. Further, we investigate the optimal quantization format for different deployments by analyzing inference performance through the popular vLLM framework. Our analysis provides clear deployment recommendations: W4A16 is the most cost-efficient for synchronous setups, while W8A8 dominates in asynchronous continuous batching. For mixed workloads, the optimal choice depends on the specific use case. Our findings offer practical, data-driven guidelines for deploying quantized LLMs at scale -- ensuring the best balance between speed, efficiency, and accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。