动态分配缓存空间，提升大模型推理的命中率和响应速度。
Recency/Frequency Adaptive KV Caching for Large Language Model Serving
- 根据访问频率和时间自适应分配缓存，避免不同任务互相干扰。
- 合成数据上缓存命中率提升10.8%，首次生成时间减少12.6%。
- 适用于多任务、批量推理，且策略可解释性强。
键值（KV）缓存是加速大语言模型推理与生成的有效技术。由于推理负载规模大且多样，传统缓存难以高效利用。现有策略采用最近最少使用（LRU）淘汰机制，但导致不同无关任务间相互冲刷缓存。为此，本文提出自适应缓存机制，动态分配缓存空间以兼顾近期与高频出现的KV块。实验表明，在合成文档问答负载下，相比基础vLLM，KV缓存命中率最高提升10.8%，首次生成时间减少12.6%；在真实对话负载中，命中率提升2.1%，首次生成时间减少2.0%。该方法在批量推理中表现良好，具备清晰可解释性，能有效适配多样化工作负载。
原文摘要 · Abstract (English)
Key-value (KV) caching is a powerful technique for accelerating large language model inference and generation. Inference workloads are large and diverse, which makes them difficult to cache effectively. Existing cache management strategies adopt the least-recently-used policy for evicting cache blocks. However, LRU leads to multiple unrelated workloads flushing each other's caches. To address this, we integrate adaptive caching that dynamically allocates cache space between recently and frequently occurring KV blocks. Evaluations show that it improves the KV cache hit rate by up to 10.8% and reduces time to first token by up to 12.6% over naive vLLM on synthetic document question answering workloads, and 2.1% and 2.0% respectively on real-world conversation workloads. The method generalizes well to batch inference and demonstrates clear interpretability while effectively accommodating diverse workloads.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。