动态压缩长文本推理的键值缓存，提速三倍且不损失关键信息。
HeteroCache: A Dynamic Retrieval Approach to Heterogeneous KV Cache Compression for Long-Context LLM Inference
- 按注意力头的稳定性与相似性分组，精细分配缓存空间。
- 在224K上下文下实现3倍解码加速，性能领先现有方法。
- 无需训练，适合长文本生成与高吞吐推理场景。
KV缓存的线性内存增长严重制约大模型在长上下文任务中的推理效率。现有静态压缩方法常丢失全局重要信息；虽有动态检索方法尝试解决，但普遍存在粗粒度缓存策略和高昂的I/O开销。为此，我们提出HeteroCache——一种无需训练的动态压缩框架。其基于两大洞察：注意力头存在显著的时间异质性，同层头间存在强空间冗余。据此，HeteroCache根据头的稳定性和相似性进行分类，采用细粒度加权策略，将更大缓存预算分配给注意力快速变化的头以捕捉上下文波动。同时，设计分层存储机制，由代表性头监控注意力漂移，触发异步按需上下文检索，从而隐藏I/O延迟。实验表明，HeteroCache在长上下文基准上达到业界最优性能，在224K上下文长度下解码速度提升最高达3倍。
原文摘要 · Abstract (English)
The linear memory growth of the KV cache poses a significant bottleneck for LLM inference in long-context tasks. Existing static compression methods often fail to preserve globally important information. Although recent dynamic retrieval approaches attempt to address this issue, they typically suffer from coarse-grained caching strategies and incur high I/O overhead. To overcome these limitations, we propose HeteroCache, a training-free dynamic compression framework. Our method is built on two key insights: attention heads exhibit diverse temporal heterogeneity, and there is significant spatial redundancy among heads within the same layer. Guided by these insights, HeteroCache categorizes heads based on stability and similarity, applying a fine-grained weighting strategy that allocates larger cache budgets to heads with rapidly shifting attention to capture context changes. Furthermore, it features a hierarchical storage mechanism where representative heads monitor attention drift to trigger asynchronous, on-demand context retrieval, thereby hiding I/O latency. Experiments demonstrate that HeteroCache achieves state-of-the-art performance on long-context benchmarks and accelerates decoding by up to $3\times$ compared to the original model with a 224K context. Our code is available at https://github.com/ponytaill/HeteroCache.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。