arXiv:2510.11292cs.LGcs.AI2025-10被引 5

通过语义边界检索与细粒度管理,提升长序列推理的KV缓存效率。

LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequences

  • 基于语义边界的动态检索,减少冗余计算
  • 输入输出分治管理,实现近无损精度下4.7倍加速
  • 适合大模型长文本生成与推理场景

尽管键值(KV)缓存能减少自回归模型中的重复计算,但其带来显著内存开销,限制了在长序列场景下的实际部署。现有方法虽通过动态保留部分KV条目缓解问题,但仍受逐令牌检索和粗粒度页级管理影响,尤其在长输出推理中效率与精度瓶颈明显。针对大模型推理需求,本文提出两个关键观察:(1)关键KV在解码过程中呈现强时间局部性;(2)输入提示与生成输出中的关键KV分布模式不同。基于此,我们提出LouisKV框架,采用语义感知检索策略,在语义边界触发检索,大幅降低计算与数据传输开销。同时设计解耦的细粒度管理机制,对输入与输出序列分别采用差异化策略,使检索单元更匹配注意力模式,精准识别关键KV。此外,通过定制Triton与CUDA内核优化,加速KV聚类与检索过程。实验表明,LouisKV在多种长序列任务中相较最先进方法最高提速4.7倍,且保持近无损精度,涵盖长输入短输出、短输入长输出及长输入长输出场景。

原文摘要 · Abstract (English)

While Key-Value (KV) cache succeeds in reducing redundant computations in auto-regressive models, it introduces significant memory overhead, limiting its practical deployment in long-sequence scenarios. Existing KV retrieval methods mitigate this by dynamically retaining only a subset of KV entries on the GPU. However, they still suffer from notable efficiency and accuracy bottlenecks due to per-token retrieval and coarse-grained page-level KV management, especially in long-output reasoning scenarios. With the emergence of large reasoning models, efficiently handling such scenarios has become increasingly important. To address this issue, we present two key observations: (1) critical KVs exhibit strong temporal locality during decoding, and (2) these KVs exhibit distinct distribution patterns across the input prompt and generated output. Building on these observations, we propose LouisKV, an efficient KV cache retrieval framework designed for various long-sequence scenarios. Specifically, LouisKV introduces a semantic-aware retrieval strategy leveraging temporal locality to trigger retrieval only at semantic boundaries, drastically reducing computation and data transfer overhead. LouisKV also designs a decoupled, fine-grained management scheme that tailors differentiated strategies for input and output sequences to create retrieval units that better match the model's attention patterns, enabling precise identification of critical KVs. Furthermore, to boost efficiency, LouisKV incorporates several kernel-level optimizations, including custom Triton and CUDA kernels to accelerate the KV clustering and retrieval. Evaluations show that LouisKV achieves up to 4.7$\times$ speedup over state-of-the-art KV retrieval methods while maintaining near-lossless accuracy across diverse long-sequence tasks, including long-input short-output, short-input long-output, and long-input long-output scenarios.

KV缓存长序列推理加速大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。