通过识别注意力头的稳定性，动态管理缓存以大幅降低大模型推理内存占用。
FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Management
- 按注意力头的时序稳定性分类，区分保留与卸载缓存页。
- 长上下文请求下显存减少70%，吞吐提升1.38-1.55倍，延迟降低1.6-2.1倍。
- 适合需要长时间生成和高并发推理的场景，如在线客服、内容创作。
大型语言模型（LLM）服务受限于随上下文长度和生成长度增长的键值（KV）缓存大小。已有研究表明，注意力机制主要依赖少数关键标记，但现有系统难以高效利用这一特性而不降低精度，尤其在长生成场景中。我们观察到：不同注意力头的关键标记具有显著不同的时序稳定性——部分头持续关注相同标记，而另一些则频繁变化。基于此，我们提出 FlexiCache，一种分层的 KV 缓存管理机制，利用注意力头的时序稳定性，在保持模型精度的前提下，降低 GPU 显存占用和计算开销。FlexiCache 将 KV 头分为稳定与不稳定两类：不稳定头的全部缓存页保留在 GPU 内存中；对稳定头，仅在 GPU 上保留前 K 个页面，其余卸载至主机内存。通过周期性重排序，为稳定头动态加载新进入前 K 的页面。基于 vLLM 实现的 FlexiCache 在长上下文请求中显存占用最高降低 70%，离线服务吞吐提升 1.38–1.55 倍，线上生成延迟降低 1.6–2.1 倍，且在长上下文、长生成任务中保持准确率。
原文摘要 · Abstract (English)
Large Language Model (LLM) serving is increasingly constrained by the growing size of the key-value (KV) cache, which scales with both context length and generation length. Prior work shows that attention is dominated by a small subset of critical tokens, yet existing systems struggle to exploit this efficiently without degrading accuracy, especially in long generation. We make a key observation: the temporal stability of these critical tokens varies significantly across KV heads: some heads consistently focus on the same tokens, while others shift frequently. Building on this insight, we introduce FlexiCache, a hierarchical KV-cache management system that leverages the temporal stability of KV heads to reduce GPU memory usage and computation overhead, while preserving model accuracy. FlexiCache classifies KV heads as stable or unstable: it retains all KV-cache pages from unstable heads in GPU memory, whereas for stable heads, it keeps only the top-K pages on the GPU and offloads the rest to host memory. By exploiting temporal stability, FlexiCache performs periodic reranking for stable heads to fetch newly promoted top pages. Implemented atop vLLM, FlexiCache reduces GPU memory footprint for long-context requests by up to 70%, improves offline serving throughput by 1.38-1.55x, and lowers online token latency by 1.6-2.1x, all while maintaining accuracy in long-context, long-generation scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。