通过追踪异常令牌提升2比特键值缓存量化精度
Accurate KV Cache Quantization with Outlier Tokens Tracing
- 识别解码中异常令牌并跳过量化
- 2比特量化下显著提升精度，内存降6.4倍
- 适合追求高吞吐与低内存的LLM部署
大型语言模型（LLMs）在推理阶段消耗大量计算资源。尽管键值缓存（KV Cache）能减少重复计算，但也会带来额外内存开销。键值缓存量化是平衡内存占用与精度的有前景方案。已有研究发现，键（Keys）按通道分布，值（Values）按标记分布，因此通常对键采用通道量化，对值采用标记量化。然而我们进一步发现，少量异常令牌表现出偏离该模式的独特特征，严重影响量化精度。为此，我们在解码过程中提出一种简单有效的方法，精准识别这些异常令牌并将其排除在量化之外，显著提升整体精度。大量实验表明，该方法在2比特量化下实现显著精度提升，可实现6.4倍内存压缩和2.3倍吞吐率增长。
原文摘要 · Abstract (English)
The impressive capabilities of Large Language Models (LLMs) come at the cost of substantial computational resources during deployment. While KV Cache can significantly reduce recomputation during inference, it also introduces additional memory overhead. KV Cache quantization presents a promising solution, striking a good balance between memory usage and accuracy. Previous research has shown that the Keys are distributed by channel, while the Values are distributed by token. Consequently, the common practice is to apply channel-wise quantization to the Keys and token-wise quantization to the Values. However, our further investigation reveals that a small subset of unusual tokens exhibit unique characteristics that deviate from this pattern, which can substantially impact quantization accuracy. To address this, we develop a simple yet effective method to identify these tokens accurately during the decoding process and exclude them from quantization as outlier tokens, significantly improving overall accuracy. Extensive experiments show that our method achieves significant accuracy improvements under 2-bit quantization and can deliver a 6.4 times reduction in memory usage and a 2.3 times increase in throughput.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。