通过三路分发提升缓存压缩效率，让可恢复信息不被丢弃。
A Simple Plug-in for Improving Eviction-Based KV Cache Compression
- 引入重要性与可重建性双信号，实现保留、近似、淘汰三路路由
- 在中高压缩率下显著改善质量-内存权衡，严格预算下提升明显
- 无需修改主模型，可直接嵌入现有淘汰式缓存压缩流程
KV缓存增长是大语言模型长上下文推理的主要瓶颈。现有方法多依赖二元淘汰或表示近似，可能浪费非关键但可重建的令牌信息。我们提出VECTOR，一种即插即用的增强方案，采用三路令牌路由：保留、近似、淘汰。该方法结合基础评分器的重要信号与离线校准的基于回归的价值估计的可重建性信号。通过利用可重建性，VECTOR恢复了原本在二元淘汰下不可逆丢失的有用价值信息，同时保持注意力路由稳定的关键词向量。实验表明，在中到高压缩率下，VECTOR显著改善了质量-内存权衡，尤其在更严格的资源预算下表现突出。
原文摘要 · Abstract (English)
KV cache growth is a major bottleneck for long-context inference in large language models. Existing methods are often dominated by binary eviction or representation approximation, which may underutilize tokens that are not critical for exact retention but are still reconstructable. We present VECTOR, a plug-and-play augmentation for eviction-based pipelines that introduces three-way token routing: retention, approximation, and eviction. VECTOR combines an importance signal from the base scorer with a reconstructability signal from an offline-calibrated regression-based value estimation. By leveraging reconstructability, VECTOR recovers useful value information that would otherwise be irreversibly lost under binary eviction, while preserving key vectors for attention routing stability. Experimental results show that VECTOR improves quality-memory trade-offs under medium-to-high compression, with especially clear gains in stricter budget regimes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。