通过异步预取键值缓存，提升大模型推理吞吐量
Accelerating LLM Inference Throughput via Asynchronous KV Cache Prefetching
- 利用空闲内存带宽预取键值缓存到L2缓存
- 使注意力计算效率提升2.15倍，端到端吞吐提高1.97倍
- 可与现有框架融合，适合大规模模型推理部署
大语言模型在推理过程中因高带宽内存（HBM）带宽受限而表现出显著的内存瓶颈。本文提出一种面向L2缓存的异步键值（KV）缓存预取方法，通过在计算活跃窗口期间调度空闲内存带宽，主动将后续所需KV缓存预取至GPU L2缓存，从而实现高速命中，有效在计算周期内隐藏HBM访问延迟。在NVIDIA H20 GPU上的大量实验表明，该方法使注意力核函数效率提升2.15倍，端到端吞吐最高提升1.97倍，超越当前最优基线FlashAttention-3。该方案与现有优化技术正交，可无缝集成至现有推理框架，为下一代大模型推理引擎提供可扩展的延迟隐藏方案。
原文摘要 · Abstract (English)
Large Language Models (LLMs) exhibit pronounced memory-bound characteristics during inference due to High Bandwidth Memory (HBM) bandwidth constraints. In this paper, we propose an L2 Cache-oriented asynchronous KV Cache prefetching method to break through the memory bandwidth bottleneck in LLM inference through computation-load overlap. By strategically scheduling idle memory bandwidth during active computation windows, our method proactively prefetches required KV Cache into GPU L2 cache, enabling high-speed L2 cache hits for subsequent accesses and effectively hiding HBM access latency within computational cycles. Extensive experiments on NVIDIA H20 GPUs demonstrate that the proposed method achieves 2.15x improvement in attention kernel efficiency and up to 1.97x end-to-end throughput enhancement, surpassing state-of-the-art baseline FlashAttention-3. Notably, our solution maintains orthogonality to existing optimization techniques and can be integrated with current inference frameworks, providing a scalable latency-hiding solution for next-generation LLM inference engines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。