通过压缩分页注意力提升大模型推理并发能力
Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttention
- 将逐标记缓存淘汰与分页注意力结合，动态管理内存
- 在数学推理任务中达全量缓存95%性能，速度提升2.1倍以上
- 适合高并发大模型服务场景，支持前缀缓存和异步压缩
随着推理成为大语言模型的主流生成范式，解码阶段的键值缓存（KV cache）内存瓶颈已成为限制高并发服务的关键因素。现有缓存淘汰方法虽能缓解内存压力，但多数难以应用于工业级系统。本文提出压缩分页注意力（Compressed PagedAttention），融合逐标记缓存淘汰与分页注意力机制，设计全面调度策略，并支持前缀缓存与异步压缩。基于此构建了高并发大模型推理引擎Zipage。在大规模数学推理任务上，Zipage实现接近全量缓存引擎95%的性能，同时获得超过2.1倍的加速比。
原文摘要 · Abstract (English)
With reasoning becoming the generative paradigm for large language models (LLMs), the memory bottleneck caused by KV cache during the decoding phase has become a critical factor limiting high-concurrency service. Although existing KV cache eviction methods address the memory issue, most of them are impractical for industrial-grade applications. This paper introduces Compressed PagedAttention, a method that combines token-wise KV cache eviction with PagedAttention. We propose a comprehensive scheduling strategy and support prefix caching and asynchronous compression for Compressed PagedAttention. Based on this, we have developed a high-concurrency LLM inference engine, Zipage. On large-scale mathematical reasoning tasks, Zipage achieves around 95\% of the performance of Full KV inference engines while delivering over 2.1$\times$ speedup.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。