Transformer推理中键值缓存可完全移除，仅靠残差流即可重建输出。
The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inference
- 所有层的键值均由残差流确定性投影生成，重计算无误差。
- 移除缓存后贪婪解码输出完全一致，跨任务验证无信息损失。
- 新方案KV-Direct仅存残差向量，内存占用降低96%以上，性能更优。
键值（KV）缓存被广泛视为Transformer推理中的必要状态，大量工作致力于压缩、淘汰或近似其条目。本文证明该状态完全冗余：每一层的键值均为残差流的确定性投影，仅需每个标记一个残差向量即可完全重构，重建误差为零，且逐位相同。我们在四个架构家族的六种模型（参数量135M至4B）上验证了这一结论。每层进行残差块拼接后，修复输出分布与原分布间互信息熵差为0（D_KL = 0），证实残差流满足马尔可夫性质，是唯一携带信息的状态。完全移除缓存并从头重计算，在所有测试模型上贪婪解码结果完全一致。基于此，我们提出KV-Direct——一种有界内存推理方案，以每标记5KB（Gemma 3-4B）的残差向量代替136KB的完整键值对，按需重计算。在超过20轮对话中，峰值内存维持在42MB，而标准缓存超103MB。相较五种淘汰策略（H2O、StreamingLLM、SnapKV、TOVA、仅窗口），在任意缓存预算下KV-Direct均保持100%标记匹配，其余方法下降至5%-28%。操作延迟分析显示，在中等批量下重计算速度比读取缓存快达5倍。代码已开源：https://github.com/Kaleemullahqasim/KV-Direct。
原文摘要 · Abstract (English)
The key-value (KV) cache is widely treated as essential state in transformer inference, and a large body of work engineers policies to compress, evict, or approximate its entries. We prove that this state is entirely redundant: keys and values at every layer are deterministic projections of the residual stream, and recomputing them from a single residual vector per token incurs exactly zero reconstruction error, not approximately, but bit-identically. We verify this across six models from four architecture families (135M to 4B parameters). Cross-task residual patching at every layer produces D_KL = 0 between patched and original output distributions, confirming that the residual stream satisfies a Markov property and is the sole information-carrying state. Removing the cache entirely and recomputing from scratch yields token-identical output under greedy decoding on all models tested. We build on this result with KV-Direct, a bounded-memory inference scheme that checkpoints residual vectors (5 KB per token on Gemma 3-4B) instead of full KV pairs (136 KB), recomputing keys and values on demand. Over 20 conversation turns, KV-Direct holds peak memory at 42 MB while the standard cache grows past 103 MB. Against five eviction baselines (H2O, StreamingLLM, SnapKV, TOVA, window-only), KV-Direct maintains 100% token match at every cache budget; all baselines degrade to 5-28%. A per-operation latency analysis shows recomputation runs up to 5x faster than reading cached tensors at moderate batch sizes. Code is available at https://github.com/Kaleemullahqasim/KV-Direct.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。