针对大模型推理的缓存压缩，实现分层差异化策略与预算分配。
PolyKV: Heterogeneous Retention and Allocation for KV Cache Compression
- 按层选择不同压缩方法，动态分配缓存预算。
- 在512令牌预算下，性能恢复率达54.5%和25.7%。
- 适合需要长文本推理的高效部署场景。
KV缓存压缩对于降低长上下文大语言模型推理的内存开销至关重要。现有方法通常对所有Transformer层采用单一压缩策略和统一缓存预算，忽略了各层在预填充和解码阶段角色差异，导致可能需要不同的淘汰策略和缓存容量。我们提出PolyKV，一种分层的KV缓存优化框架，综合考虑方法选择与预算分配的设计空间。PolyKV根据层级信号将每层路由至合适的压缩策略，并在固定总预算下分配非均匀预算。该设计实现了现有缓存方法的异构组合。在LLaMA-3.1-8B和Qwen3-8B上的实验表明，在相同512令牌平均KV缓存预算下，PolyKV分别恢复了最强单策略基线与FullKV之间54.5%和25.7%的LongBench性能差距。在128–1024的预算范围内，PolyKV始终优于最强基线1.7%–6.4%，对应恢复40.0%–54.5%的FullKV性能差距。
原文摘要 · Abstract (English)
KV cache compression is essential for reducing the memory cost of long-context large language model inference. Existing approaches, however, typically apply a single compression policy and a uniform cache budget across all transformer layers. This uniform design ignores the fact that different layers can play different roles during prefill and decoding, and may therefore require different eviction strategies and cache capacities. We present PolyKV, a layer-wise KV cache optimization framework that considers design space with method selection and budget allocation. PolyKV routes each layer to a suitable KV compression policy based on layer-level signals, while assigning non-uniform budgets under a fixed total budget. This formulation enables heterogeneous compositions of existing KV cache methods. Experiments on LLaMA-3.1-8B and Qwen3-8B show that, under the same 512-token average KV budget, PolyKV recovers 54.5% and 25.7% of the LongBench performance gap between the strongest single-policy baseline and FullKV, respectively. Across 128-1024 budget sweep, PolyKV consistently improves over the strongest baseline by 1.7%-6.4%, corresponding to 40.0%-54.5% recovery of the FullKV gap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。