动态分配大模型推理的键值缓存，提升异构内存下的加速效果
Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System
- 根据访问频率动态分配键值缓存到不同内存层级
- 理论证明可提升聚合带宽利用率，留有显著优化空间
- 适合关注大模型推理性能优化的研究者与工程师
大语言模型推理日益受制于内存带宽，频繁访问键值（KV）缓存导致大量数据移动。尽管注意力稀疏性降低了部分内存流量，但历史令牌的相关性随时间变化，仍需保持完整KV缓存可访问，持续增加带宽与容量压力。随着NVLink和LPDDR5X等互连技术发展，现代AI硬件已集成高带宽内存（HBM）与高速片外DRAM，使异构内存系统成为可行方案。本文研究在该类系统中动态调度KV缓存的放置策略，以在容量约束下最大化聚合带宽利用率。不提出特定调度策略，而是从数学上建模放置问题并推导理论上限，揭示运行时优化的巨大潜力。据我们所知，这是首次对异构内存系统中大模型推理动态KV缓存调度的正式分析。
原文摘要 · Abstract (English)
Large Language Model (LLM) inference is increasingly constrained by memory bandwidth, with frequent access to the key-value (KV) cache dominating data movement. While attention sparsity reduces some memory traffic, the relevance of past tokens varies over time, requiring the full KV cache to remain accessible and sustaining pressure on both bandwidth and capacity. With advances in interconnects such as NVLink and LPDDR5X, modern AI hardware now integrates high-bandwidth memory (HBM) with high-speed off-package DRAM, making heterogeneous memory systems a practical solution. This work investigates dynamic KV cache placement across such systems to maximize aggregated bandwidth utilization under capacity constraints. Rather than proposing a specific scheduling policy, we formulate the placement problem mathematically and derive a theoretical upper bound, revealing substantial headroom for runtime optimization. To our knowledge, this is the first formal treatment of dynamic KV cache scheduling in heterogeneous memory systems for LLM inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。