arXiv:2608.15584cs.LGcs.AI2026-08

针对多智能体长共用前缀场景,设计不对称缓存分页机制提升推理吞吐。

GraniKV: Asymmetric Granularity KV-Cache Paging for Multi-Agent Systems with Long Shared Prefix

  • 共享前缀用连续块存储,请求后缀按词元粒度分配,实现异构存储。
  • 在16K共享令牌下,吞吐最高提升2.16倍,主要来自注意力融合优化。
  • 适合处理不同长度提示的异构多智能体服务,尤其在前缀共享场景下优势明显。

生产级分页服务引擎对键值缓存采用统一分页粒度,但多智能体工作负载的两个区域存储需求相反:长共用前缀需要连续性,而每个请求的后缀则需细粒度分配。本文提出GraniKV,一种键值缓存层,将共享前缀存入连续的HOT池,后缀存入词元级的COLD池,并结合每步调度器,在双后端间动态选择适合当前计算、内存或通信瓶颈的处理路径。据我们所知,GraniKV是首个在生产级分页服务引擎中应用不对称分页粒度的系统。当共享令牌数为16,000时,相较于生产基线,GraniKV在Llama-3.1-8B/TP=1、Qwen-2.5-14B/TP=2和Qwen-2.5-32B/TP=4上分别实现2.16×、1.98×和1.57×的输出令牌吞吐提升。其中,级联注意力集成贡献了饱和状态下的主要增益;不对称存储层本身带来1.05–1.15×的端到端性能提升,并使批处理GEMM前缀后端成为可能。在具有不同长度提示的异构多智能体服务场景下,收益反转:GraniKV保持1.95×优势,而全局批处理级联注意力退化至基线水平——正是存储层带来了本论文所驱动的核心优势。

原文摘要 · Abstract (English)

Production paged-serving engines apply uniform paging granularity to the KV cache, even though the two regions of a multi-agent workload have opposite storage requirements: a long shared prefix demands contiguity, while the per-request suffix demands fine-grained allocation. We present \textbf{GraniKV}, a KV-cache layer that allocates the shared prefix in a contiguous HOT pool and the suffix in a token-level COLD pool, combined with a per-step dispatcher which selects the appropriate backend among dual backends for each regime (compute-, memory-, or communication-bound). To the best of our knowledge, GraniKV is the first system to apply asymmetric paging granularity to the KV cache of a production paged-serving engine. At $L_p{=}16$\,K shared tokens GraniKV reaches $\mathbf{2.16\times}$, $\mathbf{1.98\times}$, and $\mathbf{1.57\times}$ output-token throughput over the production baseline on Llama-3.1-8B/TP=1, Qwen-2.5-14B/TP=2, and Qwen-2.5-32B/TP=4. The gain decomposes: cascade attention integration contributes the majority at saturation; the asymmetric storage layer adds $1.05$--$1.15\times$ end-to-end while being what makes the batched-GEMM prefix backend possible at all. Under heterogeneous multi-agent serving with \emph{distinct} prompts of different lengths, the attribution inverts: GraniKV sustains $\mathbf{1.95\times}$ while batch-global cascade collapses to parity --- the storage layer alone carries the win in the regime that motivates the paper.

KV缓存多智能体分页优化推理加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。