小模型推理时复用历史键值对，提升上下文处理能力。
KV Cache Recycling to Expand Usable Context Capacity in Low Parameter LLMs
- 通过句向量匹配缓存旧输入的键值对，新输入若为前缀则复用。
- 有前缀重叠时推理速度显著提升，最长可缓存2048词元。
- 无需修改模型，适合低参数大语言模型部署优化。
小型语言模型在处理一个提示时计算出的注意力键值（KV）状态，能否被用于加速处理新相似提示？本文提出一种称为“令牌回收”的方法，通过构建过往激活状态缓存，并基于句子嵌入进行匹配，当新输入是缓存提示的精确前缀时，复用已缓存的键值对，从而扩展上下文内存空间。以Hugging Face平台上的DialoGPT-medium（345M参数，基于Reddit 2005-2017年1.47亿条对话训练）为测试基准，实验表明：在存在前缀重叠时，推理延迟显著降低，输出语义无明显退化；无重叠时行为与基线一致。整个过程无需模型修改，缓存的键值对序列化至CPU，可重新加载并注入生成函数，实现从缓存前缀继续解码。重现性良好。
原文摘要 · Abstract (English)
Whether attention key value (KV) states computed for one prompt for a small LLM can be reused to accelerate inference on a new similar prompt, giving an increase to the space to its context memory using an approach called token recycling. Using a standard Hugging Face setup with DialoGPT-medium (a 345M parameter GPT-2 style decoder trained on 147M Reddit exchanges, 2005 to 2017) as the testbed, we build a cache of past activations and get entries by sentence embeddings, then reuse cached past key values when the cached prompt is an exact prefix of the new input. We compare recycled vs. baseline runs on latency and output fidelity, and log reuse depth in tokens. Reproducibility requires no model modifications, cached KVs are serialized to the CPU, reloaded, and supplied to the generate function to continue decoding from the cached prefix. In tests, we observe consistent speedups when prefix overlap exists, with no material degradation in output semantics, and when overlap is absent, behavior matches baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。