系统梳理大模型推理中键值缓存的管理架构与评估短板
From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
- 从局部页、分布式流水线到混合层级，提炼五类缓存架构
- 指出当前评估缺少七项关键指标，影响系统可靠性判断
- 适合关注大模型部署优化与缓存设计的研究者和工程师
键值（KV）缓存已成为大模型服务中的核心内存对象，而非临时请求张量。本综述基于局部性、生命周期、所有权和载体四个维度，对三十余个KV管理机制与框架进行分类，揭示出五种架构范式：本地分页、分离流水线、共享存储、内存池和混合层级。在工作负载与硬件确定后，所有权策略成为分布式系统间设计差异的主要来源。综述还审查了现有评估方法，识别出七项缺失的KV专用测量指标，其关联于容错性、资源隔离、分层淘汰、推测解码、MoE服务及共享缓存语义等开放问题。
原文摘要 · Abstract (English)
The key-value (KV) cache has become a first-order memory object in LLM serving rather than a temporary per-request tensor. This survey classifies more than thirty KV-management systems and frameworks using four axes: locality, lifetime, ownership, and substrate. The axes reveal five architectural archetypes -- local-paged, disaggregated-pipeline, shared-store, memory-pool, and hybrid-tier. Once workload and hardware are fixed, ownership accounts for much of the remaining design variance among distributed systems. The survey also audits current evaluations and identifies seven missing KV-specific measurements, linking them to open problems in fault tolerance, isolation, tiered eviction, speculative decoding, MoE serving, and shared-cache semantics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。