用INT8量化压缩大模型推理中的键值缓存，内存减少4倍且精度几乎不变
GPU-Accelerated INT8 Quantization for KV Cache Compression in Large Language Models
- 在GPU上实现四类CUDA内核，对键值缓存进行高效INT8量化
- 内存减少4倍，8K维注意力头误差低于0.1，重建误差小于0.004
- 适合需要低延迟、高吞吐的大模型部署场景，如在线服务
大型语言模型推理中的键值（KV）缓存是显著的内存瓶颈，其大小随序列长度线性增长，常超过模型权重本身的内存占用。本文实现并评估了基于GPU的INT8量化方法用于KV缓存压缩，在保持极小精度损失的前提下实现了4倍内存缩减。我们开发了四种CUDA内核变体——朴素、分块、粗粒度和向量化，并在高达10亿元素的真实负载下进行基准测试。其中向量化内核相比CPU基线最高提升1,694倍，重建误差低于0.004，注意力分数误差低于0.1，即使在8K维头的情况下仍有效。结果表明，INT8量化是一种在计算开销仅6–58毫秒、对下游模型行为影响极小的前提下缓解大模型推理内存压力的实用方案。
原文摘要 · Abstract (English)
The key-value (KV) cache in large language models presents a significant memory bottleneck during inference, growing linearly with sequence length and often exceeding the memory footprint of model weights themselves. We implement and evaluate GPU-accelerated INT8 quantization for KV cache compression, achieving 4$\times$ memory reduction with minimal accuracy degradation. We develop four CUDA kernel variants -- naive, tiled, coarsened, and vectorized -- and benchmark them across realistic workload sizes up to 1 billion elements. Our vectorized kernel achieves up to 1,694$\times$ speedup over CPU baselines while maintaining reconstruction error below 0.004 and attention score error below 0.1 even for 8K-dimensional heads. These results demonstrate that INT8 quantization provides a practical approach for reducing memory pressure in LLM inference with negligible computational overhead (6--58ms) and minimal impact on downstream model behavior
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。