通过预测队列信息优化大模型推理缓存，显著提升响应速度和吞吐。
PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving
- 利用前缀共享结构动态管理缓存，优先保留等待请求的公共前缀
- 在五种工作负载下，最高提升3.0倍缓存命中率、7.9倍首字延迟
- 适合高并发流式推理场景，对无前缀结构任务无性能损失
本文提出PEEK，一种轻量级调度与缓存淘汰框架，适用于在线（流式）和离线（批处理）大模型服务，重点聚焦在线场景。PEEK在待处理队列上维护增量基数树，揭示现有引擎无法暴露的前缀共享聚类。通过低开销双遍历匹配树与引擎前缀缓存，为每个等待请求实现最长前缀匹配；随后优先接纳聚类首个请求，使同簇后续请求继承已缓存前缀。协同设计的淘汰钩子保护队列需求的祖先块，多车道步进调度器限制饥饿问题。在SGLang和vLLM上，针对五个工作负载（最大4×H100，DP=2，TP=2），PEEK相较各自最强基线分别实现最高3.0×/2.6×缓存命中率、7.9×/7.1×TTFT、6.7×/5.5×E2E延迟、3.6×/4.5×吞吐提升，而在无可利用前缀结构的任务上性能与基线相当。性能优势在缓存压力与推理并行度增加时依然保持。
原文摘要 · Abstract (English)
We present PEEK, a lightweight scheduling and eviction framework for both online (streaming) and offline (batch) LLM serving; this paper focuses on the online regime. PEEK maintains an incremental radix tree over the pending queue, exposing prefix-sharing clusters no existing engine surfaces. A low-overhead dual-walk matches the tree against the engine's prefix cache to yield longest-prefix-match for every waiting request; PEEK then admits cluster pioneers first so siblings inherit the freshly cached prefix, a co-designed eviction hook protects blocks ancestral to queued demand, and a multi-lane stride scheduler bounds starvation. On SGLang and vLLM across five workloads up to 4$\times$H100 (DP=2 over TP=2), PEEK delivers up to 3.0$\times$/2.6$\times$ cache hit, 7.9$\times$/7.1$\times$ TTFT, 6.7$\times$/5.5$\times$ E2E, and 3.6$\times$/4.5$\times$ throughput gains over each engine's strongest stock baseline (SGLang/vLLM), while matching baselines within noise on workloads with no exploitable prefix structure. Wins hold as KV-cache pressure and inference parallelism scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。