提出KV Packet框架，实现LLM推理中无需重计算的缓存复用。
KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs
- 将缓存文档视为不可变数据包，用轻量软令牌适配器封装
- 实验显示FLOPs接近零，TTFT更低，准确率媲美完整重计算
- 适合追求低延迟、高吞吐的LLM推理部署场景
大语言模型（LLMs）严重依赖键值（KV）缓存以降低推理延迟。然而，标准KV缓存具有上下文依赖性：在新上下文中复用已缓存文档时，需重新计算KV状态以适应注意力分布的变化。现有方法如CacheBlend、EPIC和SAM-KV通过选择性重计算部分标记缓解此问题，但仍带来显著计算开销（FLOPs）和更高的首次生成时间（TTFT）。本文提出KV Packet，一种无需重计算的缓存复用框架，将缓存文档视为不可变的“数据包”，并用轻量可训练的软令牌适配器包裹，通过自监督蒸馏训练以弥合上下文断层。在Llama-3.1和Qwen2.5上的实验表明，该方法实现接近零的FLOPs，TTFT低于基于重计算的基线，同时保持与完整重计算基线相当的F1得分。
原文摘要 · Abstract (English)
Large Language Models (LLMs) rely heavily on Key-Value (KV) caching to minimize inference latency. However, standard KV caches are context-dependent: reusing a cached document in a new context requires recomputing KV states to account for shifts in attention distribution. Existing solutions such as CacheBlend, EPIC, and SAM-KV mitigate this issue by selectively recomputing a subset of tokens; however, they still incur non-negligible computational overhead (FLOPs) and increased Time-to-First-Token (TTFT) latency. In this paper, we propose KV Packet, a recomputation-free cache reuse framework that treats cached documents as immutable ``packets'' wrapped in light-weight trainable soft-token adapters, which are trained via self-supervised distillation to bridge context discontinuities. Experiments on Llama-3.1 and Qwen2.5 demonstrate that the proposed KV Packet method achieves near-zero FLOPs and lower TTFT than recomputation-based baselines, while retaining F1 scores comparable to those of the full recomputation baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。