将大模型的键值缓存压缩至原大小2%以下，且不损失性能。
Lossless KV Cache Compression to 2%
- 提出跨层潜在注意力架构，融合多维压缩技术。
- 在多数任务上实现2%缓存占用下的无损性能。
- 适合需要高效推理的大模型部署场景。
大型语言模型在多个领域革新了数据处理，其处理长上下文推理的能力备受关注。为加速推理，保持键值（KV）缓存内存至关重要。然而，对KV缓存内存日益增长的需求给高效实现带来了显著挑战。本文提出一种新架构——跨层潜在注意力（CLLA），旨在将KV缓存压缩至原始大小的2%以下，同时保持相当的性能水平。CLLA将注意力头/维度压缩、层共享和量化技术整合到统一框架中。大量实验表明，CLLA在多数任务上实现无损性能，同时仅使用极小的KV缓存，标志着实际KV缓存压缩的重大进展。
原文摘要 · Abstract (English)
Large language models have revolutionized data processing in numerous domains, with their ability to handle extended context reasoning receiving notable recognition. To speed up inference, maintaining a key-value (KV) cache memory is essential. Nonetheless, the growing demands for KV cache memory create significant hurdles for efficient implementation. This work introduces a novel architecture, Cross-Layer Latent Attention (CLLA), aimed at compressing the KV cache to less than 2% of its original size while maintaining comparable performance levels. CLLA integrates multiple aspects of KV cache compression, including attention head/dimension reduction, layer sharing, and quantization techniques, into a cohesive framework. Our extensive experiments demonstrate that CLLA achieves lossless performance on most tasks while utilizing minimal KV cache, marking a significant advancement in practical KV cache compression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。