用位置信息模拟生成查询,精准压缩大模型缓存。
Where Matters More Than What: Decoding-aligned KV Cache Compression via Position-aware Pseudo Queries
- 基于位置伪查询模拟生成过程,构建更准确的缓存评估窗口。
- 在3%缓存预算下实现99.5%几乎无损性能(NIAH数据集)。
- 特别适合内存受限场景,如边缘部署或长文本生成。
键值(KV)缓存对大语言模型高效推理至关重要,但超长上下文会显著增加其内存开销。现有压缩方法依赖提示词窗口内的输入侧注意力模式来估计令牌重要性,却无法保留未来生成所需的关键令牌,因其评估未基于解码过程。直觉上,有效的观察窗口应反映解码阶段的查询。然而,推理时真实解码查询不可用。我们发现,位置信息比语义内容更具关键作用。据此提出DapQ:一种轻量级缓存淘汰框架,通过位置感知伪查询模拟输出令牌,建立贴近真实生成上下文的评估窗口,实现精准令牌剔除。跨多个基准和模型的实验证明,该方法性能优异,尤其在严苛内存约束下表现突出,如在NIAH数据集上仅用3%缓存预算即达99.5%近乎无损性能。
原文摘要 · Abstract (English)
The Key-Value (KV) cache is crucial for efficient Large Language Models (LLMs) inference, but excessively long contexts drastically increase KV cache memory footprint. Existing KV cache compression methods typically rely on input-side attention patterns within a prompt observation window to estimate token importance during the prefill stage. They fail to preserve critical tokens for future generation since these assessments are not derived from the decoding process. Intuitively, an effective observation window should mirror the decoding-stage queries to accurately reflect which tokens the generation process will attend to. However, ground-truth decoding queries are inherently unavailable during inference. For constructing pseudo queries to approximate them, we find that positional information plays a more critical role than semantic content. Motivated by this insight, we propose decoding-aligned KV cache compression via position-aware pseudo queries (DapQ), a novel and lightweight eviction framework that leverages position-aware pseudo queries to simulate the output tokens, thereby establishing an effective observation window for importance assessment. It aligns closely with the actual generation context and enables precise token eviction. Extensive evaluations across multiple benchmarks and LLMs demonstrate that DapQ achieves superior performance, particularly under strict memory constraints (e.g., up to nearly lossless performance 99.5% on NIAH with 3% KV cache budgets).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。