arXiv:2606.13126cs.LGcs.AI2026-06

用少于100行代码实现灵活的无位置依赖缓存,显著提升重复输入的预填充效率。

MiniPIC: Flexible Position-Independent Caching in <100LOC

论文配图:MiniPIC: Flexible Position-Independent Caching in <100LOC
图 1 · 摘自论文原文
  • 摒弃位置编码,将未旋转的键向量存入缓存,按请求逻辑位置动态应用RoPE。
  • 在2WikiMultihopQA上,预填充吞吐提升49%,首次生成延迟降低近两个数量级。
  • 支持多种缓存策略共存,适合需要高效处理重复结构化输入的推理系统。

检索增强与代理类任务频繁预填充可预测的结构化输入(称为“跨度”),如文档和代码文件。现有vLLM中的前缀缓存仅当请求前缀完全相同时才能复用KV缓存;而生产级的无位置依赖缓存(PIC)通常需大量服务端代码修改或将KV状态外置,带来主机到设备的数据传输开销。本文提出极简型无位置依赖缓存(MiniPIC):一种仅需约100行核心代码改动、基于无位置编码的KV缓存与用户可控缓存复用原语的vLLM轻量设计。它将未旋转的键向量存入缓存,在注意力计算中按请求逻辑位置动态应用RoPE,并提供块对齐填充、跨度分隔符(SSep)和提示依赖(PDep)三个细粒度控制原语,可灵活调整哈希行为与块级因果注意力结构。这些原语足以在同一vLLM实例中实现Block-Attention、EPIC与提示缓存等多种方法,且天然兼容KV缓存的CPU卸载。在2WikiMultihopQA测试中,结合交错调度的MiniPIC使预填充吞吐提升49%,缓存跨度的时延至首次输出降低近两个数量级,保持未缓存跨度的线性扩展性,最坏情况仅增加5.7%开销。

原文摘要 · Abstract (English)

Retrieval-augmented and agentic workloads repeatedly prefill recurring predictable structured inputs (which we call "spans") such as documents and code files. Yet, prefix caching in engines such as vLLM cannot reuse their KV entries unless they share identical prefixes with another request, while Position-Independent Caching (PIC) implementations within production-grade inference servers typically either require substantial server code changes or keep KV state outside the server, incurring host-to-device transfer overhead. We present Minimalistic PIC (MiniPIC): a minimal, flexible and fast vLLM design built from two ingredients: positional-encoding-free KV cache and user-controlled cache-reuse primitives. MiniPIC stores unrotated K vectors in the KV cache, applies RoPE to K tiles inside attention using per-request logical positions, and exposes three user-facing and token-level primitives: block-aligned padding, span separator (SSep), and prompt depend (PDep), that modify hashing behavior and effective block-level causal attention structure. With fewer than 100 lines of core-engine changes plus a custom attention backend, these primitives are sufficient to realize multiple PIC methods, including Block-Attention, EPIC, and Prompt Cache, within the same running vLLM instance, while natively integrating with KV cache CPU offload implementations. On 2WikiMultihopQA, MiniPIC with interleaved scheduling improves prefill throughput by 49% over baseline vLLM, reduces cached-span time-to-first-token by up to two orders of magnitude, preserves the linear prefill scaling of uncached spans, and incurs only 5.7% worst-case overhead.

缓存优化推理加速vLLMKV缓存

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。