通过查询向量几何特性，实现无需训练的高效键值缓存压缩。
Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compression
- 基于查询与键的几何特性，用单次投影筛选关键键值对。
- 在长文本任务中，压缩32倍仍保持99%准确率，生成困惑度下降65%。
- 兼容FlashAttention，适合大模型长序列生成场景使用。
自回归语言模型依赖键值（KV）缓存避免重复计算历史隐藏状态，提升生成速度。随着模型规模和上下文长度增长，KV缓存成为主要内存瓶颈，亟需压缩方法。本文发现查询（Q）与键（K）向量存在意外几何特性，可无需计算注意力图直接近似注意力分数。提出Q-Filters：一种无需训练的KV缓存压缩方法，通过单一上下文无关投影筛选不重要键值对。与多数方法不同，Q-Filters兼容FlashAttention，无需访问注意力权重。实验表明，在长上下文设置下，其在检索任务中性能媲美基于注意力的压缩方法SnapKV，且在生成场景中持续优于Streaming-LLM等高效压缩方案。尤其在针堆任务中达99%准确率，压缩32倍时生成困惑度下降最高达65%。
原文摘要 · Abstract (English)
Autoregressive language models rely on a Key-Value (KV) Cache, which avoids re-computing past hidden states during generation, making it faster. As model sizes and context lengths grow, the KV Cache becomes a significant memory bottleneck, which calls for compression methods that limit its size during generation. In this paper, we discover surprising properties of Query (Q) and Key (K) vectors that allow us to efficiently approximate attention scores without computing the attention maps. We propose Q-Filters, a training-free KV Cache compression method that filters out less crucial Key-Value pairs based on a single context-agnostic projection. Contrarily to many alternatives, Q-Filters is compatible with FlashAttention, as it does not require direct access to attention weights. Experimental results in long-context settings demonstrate that Q-Filters is competitive with attention-based compression methods such as SnapKV in retrieval tasks while consistently outperforming efficient compression schemes such as Streaming-LLM in generation setups. Notably, Q-Filters achieves a 99% accuracy in the needle-in-a-haystack task with a x32 compression level while reducing the generation perplexity drop by up to 65% in text generation compared to Streaming-LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。