KVComp通过针对性压缩技术，大幅降低大模型推理时的显存占用。
KVComp: A High-Performance, LLM-Aware, Lossy Compression Framework for KV Cache
- 针对KV缓存特性设计新型有损压缩算法，软硬协同优化。
- 平均内存压缩率提升47%，最高达83%，精度几乎无损失。
- 适合长文本生成场景，尤其对延迟和吞吐敏感的系统
基于Transformer的大语言模型在实际应用中展现出巨大潜力，但长序列推理面临巨大内存挑战，因键值（KV）缓存随序列长度和批量增大可增至数GB。本文提出KVComp，一种通用高效的KV缓存管理框架，专为长文本生成设计，兼容低延迟与高吞吐推理系统。该框架采用针对KV缓存数据特征定制的新型有损压缩技术，实现压缩算法与系统架构的协同设计。方法保持了KV缓存的动态增长特性，同时保证高计算效率。实验表明，相比现有方法，KVComp平均内存压缩率提升47%，最高达83%，模型精度几乎无损失。此外，其执行吞吐极高，有效降低解压开销，某些情况下甚至加速矩阵-向量乘法运算，并在更少数据搬移的前提下超越cuBLAS基注意力核性能。
原文摘要 · Abstract (English)
Transformer-based large language models (LLMs) demonstrate impressive potential in various practical applications. However, long context inference poses a significant challenge due to the enormous memory requirements of the key-value (KV) cache, which can scale to multiple gigabytes as sequence length and batch size increase. In this paper, we present KVComp, a generic and efficient KV cache management framework optimized for long-text generation that synergistically works with both latency-critical and throughput-critical inference systems. KVComp employs novel lossy compression techniques specifically designed for KV cache data characteristics, featuring careful co-design of compression algorithms and system architecture. Our approach maintains compatibility with the growing nature of KV cache while preserving high computational efficiency. Experimental results show that KVComp achieves on average 47\% and up to 83\% higher memory reduction rate compared to existing methods with little/no model accuracy degradation. Furthermore, KVComp achieves extremely high execution throughput, effectively reducing decompression overhead and, in some cases, even accelerating the matrix-vector multiplication operation and outperform cuBLAS-based attention kernels with less data movement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。