分析大模型服务中键值缓存使用模式，提出自适应优化策略提升性能。
KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider
- 通过真实服务数据刻画键值缓存的请求重用规律。
- 发现单轮与多轮请求的缓存重用率相当，且模式可预测。
- 设计工作负载感知的缓存淘汰策略，适配有限缓存容量场景。
大语言模型（LLM）的服务对云服务商至关重要，缓存每次请求处理后的中间结果（KV缓存）可显著提升服务吞吐量和降低延迟。然而，现有研究对实际场景下KV缓存如何提升服务性能仍缺乏系统理解，且缓存淘汰策略高度依赖具体工作负载。本文首次对某领先LLM服务提供商的真实工作负载进行了系统性分析，揭示了以往基于合成数据研究未涵盖的规律：不同请求间的KV缓存重用存在偏斜分布，单轮请求的重用与多轮请求同等重要；整体重用时间和概率差异大，但特定请求类别内的模式具有可预测性；实现理想缓存命中率所需的总缓存空间适中。基于此，我们提出一种工作负载感知的缓存淘汰策略，在真实工作负载追踪下显著提升了服务性能，尤其在缓存容量受限时效果更佳。
原文摘要 · Abstract (English)
Serving large language models (LLMs) is important for cloud providers, and caching intermediate results (KV\$) after processing each request substantially improves serving throughput and latency. However, there is limited understanding of how LLM serving benefits from KV\$ caching, where system design decisions like cache eviction policies are highly workload-dependent. In this paper, we present the first systematic characterization of the KV\$ workload patterns from one of the leading LLM service providers. We draw observations that were not covered by previous studies focusing on synthetic workloads, including: KV\$ reuses are skewed across requests, where reuses between single-turn requests are equally important as multi-turn requests; the reuse time and probability are diverse considering all requests, but for a specific request category, the pattern tends to be predictable; and the overall cache size required for an ideal cache hit ratio is moderate. Based on the characterization, we further propose a workload-aware cache eviction policy that improves the serving performance under real-world traces, especially with limited cache capacity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。