通过多层级内存与智能预取，显著提升大模型推理时的缓存效率。
Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference
- 按注意力类型精准计算内存需求，避免过度分配
- 六级内存架构使缓存容量扩展至38TB以上，保持毫秒级响应
- 基于贝叶斯预测的智能淘汰机制，命中率达70%以上
键值（KV）缓存管理是大规模GPU推理服务中限制吞吐量和成本效率的主要瓶颈。现有系统存在三大累积低效：(1) 缺乏跨所有注意力架构的统一缓存尺寸，尤其对多头潜在注意力（MLA）支持不足，导致最高达57倍的内存过度分配；(2) KV缓存仅限于单一内存层级（GPU HBM），而未利用从CPU DRAM、CXL连接内存、通过GPUDirect Storage的NVMe、RDMA网络到并行文件系统的丰富层次结构；(3) 反应式淘汰策略丢弃可复用状态，迫使重复计算。本文提出统一系统解决上述问题：架构感知的尺寸计算引擎精确确定每种注意力类型的内存需求；在评估的单个MLA模型（DeepSeek-V3）上，批处理规模提升7.4倍，而三个GQA模型分别提升1.0倍、1.0倍和0.7倍，说明收益主要来自全集群统一尺寸而非单模型更大批次。六级内存层次结构将每节点有效缓存容量从40GB扩展至超过38TB，同时维持热数据的亚毫秒级首字延迟（TTFT）。基于贝叶斯重用预测器（对16种（块类型，转移类型）组合使用共轭先验）驱动的EMA评分头粒度淘汰与RoPE感知预取，在ShareGPT、LMSYS-Chat-1M及代理工作负载的追踪回放验证中实现70%-84%的缓存命中率。结合已验证组件行为与公开硬件规格的分析投影显示，首字延迟降低1.4至2.1倍，吞吐量提升1.7至2.9倍，成本降低47%，相较已有基线；这些集群级预测为分析性结果，无误差范围。
原文摘要 · Abstract (English)
Key-value (KV) cache memory management is the primary bottleneck limiting throughput and cost-efficiency in large-scale GPU inference serving. Current systems suffer from three compounding inefficiencies: (1) the absence of unified KV cache sizing across all attention architectures--particularly multi-head latent attention (MLA), which is unsupported in general-purpose frameworks, resulting in up to 57x memory over-provisioning; (2) confinement of KV cache to a single memory tier (GPU HBM) despite the availability of a rich hierarchy spanning CPU DRAM, CXL-attached memory, NVMe via GPUDirect Storage, RDMA fabric, and parallel filesystems; and (3) reactive eviction policies that discard reusable state, forcing redundant recomputation. We present a unified system addressing all three. Our architecture-variant-aware sizing engine computes exact memory requirements per attention type; the resulting batch size gain reaches 7.4x for the one MLA model we evaluate (DeepSeek-V3), while the three GQA models see 1.0x, 1.0x, and 0.7x, so the GQA benefit is fleet-wide unified sizing rather than larger per-model batches. A six-tier memory hierarchy extends effective KV cache capacity from 40 GB to over 38 TB per node while maintaining sub-millisecond time-to-first-token (TTFT) for hot entries. A Bayesian reuse predictor with Beta conjugate priors over 16 (block-type, transition-type) pairs drives EMA-scored head-granular eviction and RoPE-aware prefetching. Component-level validation on trace replay using ShareGPT, LMSYS-Chat-1M, and agentic workloads demonstrates 70-84% cache hit rates. Analytical projections combining validated component behavior with published hardware specifications indicate TTFT reductions of 1.4x to 2.1x, throughput improvements of 1.7x to 2.9x, and 47% cost reduction relative to published baselines; these cluster-scale projections are analytical and carry no error bars.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。