KV缓存压缩在复杂指令场景下可能让模型忽略关键指令。
The Pitfalls of KV Cache Compression
- 测试多种压缩方法在多指令提示下的表现，发现部分指令退化严重。
- 压缩导致系统提示泄露，关键指令被模型完全忽略。
- 提出优化缓存淘汰策略，提升多指令任务的稳定性与准确性。
KV缓存压缩能显著提升推理吞吐量且性能损失微小，但其在真实场景（如多指令提示）中的影响尚未充分研究。本文评估了五种压缩方法（StreamingLLM、SnapKV、TOVA、H2O、K-Norm）在Llama3.1 8B和Qwen2.5 14B上的表现，使用IFEval数据集进行多指令提示测试。结果表明，某些指令在压缩下退化剧烈，甚至被模型完全忽略。以系统提示泄露为例，实证显示压缩加剧了该问题。我们识别出三个影响因素：压缩方法、指令顺序及KV缓存驱逐偏差。最后提出改进缓存淘汰策略的简单方案，有效缓解上述问题，提升多指令任务整体性能。
原文摘要 · Abstract (English)
KV cache compression promises increased throughput and efficiency with negligible loss in performance. While the gains in throughput are indisputable and recent literature has indeed shown minimal degradation on particular benchmarks, in general the consequences of compression in realistic scenarios such as multi-instruction prompting have been insufficiently studied. In this paper, we identify several pitfalls that practitioners should be aware of when deploying KV cache compressed LLMs. We evaluate five KV cache compression methods (StreamingLLM, SnapKV, TOVA, H2O, and K-Norm) on Llama3.1 8B and Qwen2.5 14B under multi-instruction prompting with IFEval. Importantly, we show that certain instructions degrade much more rapidly with compression, effectively causing them to be completely ignored by the LLM. As a practical example, we highlight system prompt leakage as a case study, empirically demonstrating the impact of compression on leakage and general instruction-following. We identify several factors that contribute to system prompt leakage: compression method, instruction order, and KV eviction bias. We then propose simple changes to KV cache eviction policies that can reduce the impact of these factors and improve the overall performance in multi-instruction tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。