用稀疏表示压缩大模型注意力缓存，1比特存储仍保持性能
CSR:Achieving 1 Bit Key-Value Cache via Sparse Representation
- 将密集的键值缓存转为稀疏索引与权重，实现高效压缩
- 在有限内存下表现稳定，性能接近顶尖量化方法
- 自动生成词典的神经网络，提升压缩自动化程度
大型语言模型（LLMs）在长文本应用中面临显著的可扩展性挑战，尤其是内存占用问题。负责存储注意力键值（KV cache）以减少重复计算的缓存呈线性增长，导致内存消耗大幅增加，可能在资源受限环境下使模型无法服务。为此，我们提出一种名为缓存稀疏表示（CSR）的新方法，将密集的键值缓存张量转换为稀疏索引和权重，实现更高效的内存表示。此外，我们引入NeuralDict，一种基于神经网络的自动词典生成方法。大量实验表明，CSR在内存受限环境中表现稳健，性能可媲美当前最先进的键值缓存量化算法。
原文摘要 · Abstract (English)
The emergence of long-context text applications utilizing large language models (LLMs) has presented significant scalability challenges, particularly in memory footprint. The linear growth of the Key-Value (KV) cache responsible for storing attention keys and values to minimize redundant computations can lead to substantial increases in memory consumption, potentially causing models to fail to serve with limited memory resources. To address this issue, we propose a novel approach called Cache Sparse Representation (CSR), which converts the KV cache by transforming the dense Key-Value cache tensor into sparse indexes and weights, offering a more memory-efficient representation during LLM inference. Furthermore, we introduce NeuralDict, a novel neural network-based method for automatically generating the dictionary used in our sparse representation. Our extensive experiments demonstrate that CSR achieves performance comparable to state-of-the-art KV cache quantization algorithms while maintaining robust functionality in memory-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。