简单量化方法RTN在大模型推理中表现不俗，可兼顾速度与精度。
Is (Selective) Round-To-Nearest Quantization All You Need?
- 采用基于Marlin内核的选择性精度提升策略优化RTN
- RTN生成吞吐量优于先进方法，精度相当
- 适合追求高效部署的工程团队参考
量化已成为服务日益增长的大语言模型（LLMs）的必要工具。RTN（四舍五入到最近）是最简单的量化技术之一，早在大模型兴起前就已存在。然而，近期更先进的量化方法普遍宣称在性能上全面超越RTN，使其被广泛忽视。本文旨在挑战这一观点，表明RTN不仅实现成本极低，且在生成吞吐量上可优于甚至媲美先进方法，同时保持相近精度。我们基于最新Marlin内核实现RTN，并展示通过有选择地提升部分模型层和模块的数据精度格式，可逐步改善RTN的精度表现。结果表明，RTN是量化LLMs的一种可行且实用的选择。
原文摘要 · Abstract (English)
Quantization became a necessary tool for serving ever-increasing Large Language Models (LLMs). RTN (Round-to-Nearest) is perhaps the simplest quantization technique that has been around well before LLMs surged to the forefront of machine learning (ML) research. Yet, it has been largely dismissed by recent and more advanced quantization methods that claim superiority over RTN in nearly every aspect of performance. This work aims to dispel this established point of view, showing that RTN is not only much cheaper to apply, but also its token generation throughput can be better than and accuracy can be similar to more advanced alternatives. In particular, we discuss our implementation of RTN based on the recent Marlin kernels and demonstrate how the accuracy of RTN can be gradually improved by selectively increasing the data precision format of certain model layers and modules. Based on our results, we argue that RTN presents a viable and practical choice for quantizing LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。