通过分层管理键值缓存，显著降低大模型响应延迟。
LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management
- 按模型层数精细分配和调度键值缓存块
- 在70B模型上将首令牌延迟降低69倍
- 适合高并发大模型服务场景
大型语言模型（LLM）上下文窗口的扩展极大提升了其应用能力，但也带来了显著的低延迟挑战，尤其在首令牌时间（TTFT）方面。本文发现，随着上下文长度增加，TTFT急剧上升主要由排队延迟引起，根源在于GPU键值（KV）缓存分配需求激增与缓存块资源有限之间的冲突。为此，我们提出LayerKV，一种简单而高效的即插即用方法，可在不增加硬件、不牺牲输出性能的前提下有效降低TTFT，且能无缝集成现有并行策略和调度技术。具体而言，LayerKV引入了分层的KV缓存块分配、管理和卸载机制，实现对系统内存的细粒度控制，并结合SLO感知调度器优化整体服务质量目标（SLO）。在从7B到70B参数的代表性模型上，覆盖多种GPU配置的全面评估表明，LayerKV可将TTFT延迟最高降低69倍，同时将SLO违反率降低28.7%，显著提升用户体验。
原文摘要 · Abstract (English)
The expanding context windows in large language models (LLMs) have greatly enhanced their capabilities in various applications, but they also introduce significant challenges in maintaining low latency, particularly in Time to First Token (TTFT). This paper identifies that the sharp rise in TTFT as context length increases is predominantly driven by queuing delays, which are caused by the growing demands for GPU Key-Value (KV) cache allocation clashing with the limited availability of KV cache blocks. To address this issue, we propose LayerKV, a simple yet effective plug-in method that effectively reduces TTFT without requiring additional hardware or compromising output performance, while seamlessly integrating with existing parallelism strategies and scheduling techniques. Specifically, LayerKV introduces layer-wise KV block allocation, management, and offloading for fine-grained control over system memory, coupled with an SLO-aware scheduler to optimize overall Service Level Objectives (SLOs). Comprehensive evaluations on representative models, ranging from 7B to 70B parameters, across various GPU configurations, demonstrate that LayerKV improves TTFT latency up to 69x and reduces SLO violation rates by 28.7%, significantly enhancing the user experience.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。