让模型笔记可编辑可组合，加速推理且保持准确
Models Take Notes at Prefill: KV Cache Can Be Editable and Composable
- 将预填充缓存视为可修改的笔记，支持直接编辑关键字段
- 编辑后仅需1%计算量即可恢复正确决策，延迟降低14.9倍
- 适用于多种模型架构，兼容现有缓存机制，适合高并发场景
前缀缓存仅在完全相同的前缀间复用，一旦字段变更，整个下游缓存失效。然而，仅替换字段对应的键值向量却会引发模型继续依赖旧值。因果分析表明：在预填充阶段，模型已将该字段决定的结论写入下游‘笔记’中，而该字段自身的键值仅贡献不到1%的决策。将此理解为记忆化的笔记，衍生出两项能力：(1) 可编辑性——通过显著修正错误注释，结合思维链（CoT）可仅通过编辑字段即恢复正确决策（8B模型达1.00准确率，仅需约1%计算量），无CoT则被忽略；(2) 可组合性——笔记具有位置可移植性，预编译技能可通过RoPE重定位并插入任意上下文，与完整重新计算效果几乎一致（对数余弦相似度0.90–0.999，覆盖十二个模型），时间复杂度从O(L²)降至O(L)。统一编辑+组合代理在延迟上比重新计算低至14.9倍，且保持决策一致性。该方法适用于任意逐标记注意力的键值缓存，已在不同规模、量化、专家混合模型及多模态缓存中验证，并通过小适配器扩展至多种注意力变体。由于纠错为追加式，可与生产级前缀缓存兼容：在在线vLLM基准测试中，保持98.5%的命中率，将p90首字延迟降低53至398倍。
原文摘要 · Abstract (English)
Prefix caching reuses prefill only across an exactly shared prefix, so one changed field invalidates the entire downstream cache. Yet overwriting the field's own key/value vectors and reusing the rest leaves the model acting on the old value. The reason, established causally across four model families: at prefill the model has already written the field-conditioned conclusion onto downstream notes; the field's own key/value drives under 1% of the decision. Read as a notebook of memoized conclusions, two capabilities follow. (1) It is editable. A salient erratum amends the notes; and with chain-of-thought, editing the field alone recovers the decision (1.00 at 8B, ~1% compute), while without CoT it is ignored. (2) It is composable. The notes are position-portable, so a precompiled skill can be RoPE-repositioned and spliced into any context, indistinguishable from full recompute (logit cosine 0.90-0.999, twelve models) at O(L) rather than O(L^2) time-to-first-token. A unified edit+compose agent stays decision-identical to recompute at up to 14.9x lower latency. The approach applies to any per-token attention KV cache, validated across scale, quantization, Mixture-of-Experts, and multimodal caches, and extends to several attention variants through small adapters. Because the erratum is append-only, it composes with production prefix caching: in an online vLLM benchmark it keeps the prefix cache-aligned (98.5% hit-rate), cutting p90 time-to-first-token by 53-398x.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。