通过比较键值缓存间的时间滞后，高效识别重要词元
LagKV: Lag-Relative Information of the KV Cache Tells Which Tokens Are Important
- 仅通过比较KV缓存自身实现压缩，无需注意力计算
- 在64位密钥检索任务中，性能优于基于注意力的方法50%以上
- 无需修改推理架构，适合主流平台快速集成
大型语言模型长文本推理过程中，键值（KV）缓存持续增长，成为部署成本与任务精度之间的主要瓶颈。现有方法多依赖注意力权重剔除非关键缓存词元，但通常需大幅修改推理系统并带来显著计算开销。鉴于大模型的自回归特性，本文提出LagKV，一种仅基于KV缓存之间直接比较的压缩策略，完全不依赖注意力机制。该方法可无缝集成至主流推理平台，性能媲美复杂压缩方法。RULER基准测试显示，本方法在不同压缩比下均优于SnapKV和StreamingLLM。尤其在64位密钥检索任务中，相比基于注意力的H₂O方法，在相同压缩比下性能提升超50%。代码已开源。
原文摘要 · Abstract (English)
The increasing size of the Key-Value (KV) cache during the Large Language Models long-context inference is the main obstacle for its balance between the deployment cost and task accuracy. To reduce the KV cache size in such scenarios, most previous efforts leveraged on the attention weight to evict non-critical cache tokens. But there is a trade-off in those methods, they usually require major modification of the inference infrastructure and significant computation overhead. Based on the fact that the Large Language models are autoregressive models, we propose LagKV, a KV compression strategy only relying on straight forward comparison among KV themselves. It is a totally attention free method which offers easy integration to the main stream inference platform and comparable performance comparing to other complicated KV compression methods. Results on RULER benchmark show that, our approach outperforms SnapKV and StreamingLLM in different compression ratios. Especially in the 64-digit passkey retrieval task, our method outperforms the attention weight based method $H_2O$ over $50\%$ with same compression ratios. Our code is available at https://github.com/AI-Lab-China-Merchants-Bank/LagKV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。