arXiv:2605.12471cs.LGcs.AI2026-05

无需训练,用折叠机制实现超长上下文推理。

KV-Fold: One-Step KV-Cache Recurrence for Long-Context Inference

  • 将键值缓存当作累加器,逐块处理长文本。
  • 在128K token上下文中实现100%准确检索,保持内存可控。
  • 适用于所有预训练模型,无需修改架构或重新训练。

我们提出KV-Fold,一种简单、无需训练的长上下文推理方法,将键值(KV)缓存视为对序列块的左折叠累加器。每一步中,模型基于累积的缓存处理下一个块,追加新生成的键值,并将扩展后的缓存传递下去;这一单步更新重复进行,类似函数式编程中的foldl操作。基于为潜在多智能体通信设计的KV缓存拼接原语,我们将之重构为块间递归机制。处理第t个块时,模型可访问前序块的缓存作为前缀,跨段复用内部状态,不修改也不重新训练模型。尽管方法简单,其诱导的递归具有稳定性:每步漂移短暂上升后趋于平缓并长期维持。该平台对数值精度变化(10,000倍)不敏感,对块大小和模型族均鲁棒。任务层面,KV-Fold能精确保留远距离信息。在针在草堆中基准测试中,它在16K至128K token上下文、链深达511的情况下,于Llama-3.1-8B上实现了152次试验中100%的精确匹配检索,且仅需单张40GB GPU内存。相比流式方法(以牺牲保真度换取有限内存),KV-Fold在保持长程检索能力的同时,仍以一系列可处理的前向传播完成。结果表明,冻结的预训练变换器已具备稳定的关键值缓存递归能力,为无需结构变更或训练的长上下文推理提供了实用路径。

原文摘要 · Abstract (English)

We introduce KV-Fold, a simple, training-free long-context inference protocol that treats the key-value (KV) cache as the accumulator in a left fold over sequence chunks. At each step, the model processes the next chunk conditioned on the accumulated cache, appends the newly produced keys and values, and passes the enlarged cache forward; the same one-step update is applied repeatedly, analogous to foldl in functional programming. Building on the KV cache concatenation primitive introduced for latent multi-agent communication, we repurpose it as a chunk-to-chunk recurrence for long-context inference. When processing chunk t, the model attends to the KV cache carried from earlier chunks as a prefix, reusing its internal state across segments without modifying or retraining the model. Despite its simplicity, the induced recurrence is stable: per-step drift rises briefly and then saturates into a flat plateau that persists across deep chains. This plateau is insensitive to a 10,000x change in numerical precision, robust across chunk sizes, and consistent across model families. At the task level, KV-Fold preserves exact information over long distances. On a needle-in-a-haystack benchmark, it achieves 100% exact-match retrieval across 152 trials spanning contexts from 16K to 128K tokens and chain depths up to 511 on Llama-3.1-8B, while remaining within the memory limits of a single 40GB GPU. Compared to streaming methods, which trade fidelity for bounded memory, KV-Fold maintains long-range retrieval while operating as a sequence of tractable forward passes. Overall, our results show that frozen pretrained transformers already support a stable form of KV-cache recurrence, providing a practical route to long-context inference without architectural changes or training.

长上下文推理优化KV缓存无训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。