arXiv:2608.15797cs.AIcs.CL2026-08

通过融合完整上下文模型,缓解长推理中因缓存丢失导致的精度下降问题。

KV-Rescue: Recovering Reasoning Language Model KV Eviction Loss via Stepwise Interleaving

论文配图:KV-Rescue: Recovering Reasoning Language Model KV Eviction Loss via Stepwise Interleaving
图 1 · 摘自论文原文
  • 用轻量级全上下文模型与主模型交替生成,填补缓存缺失的信息空缺。
  • 在五项数学基准上恢复87%的精度损失,且平均减少43%无效生成。
  • 无需训练即可应用,特别适合资源受限的长序列推理场景。

KV缓存淘汰限制了长推理过程的内存开销,但本质上是信息丢失的,因为模型仅基于部分历史进行解码。在严格预算下,这不仅降低准确率,还可能导致失控退化(模型生成不连贯或重复的标记直至达到长度上限)。我们发现这种损失主要源于缺失上下文的信息缺口,而非模型能力不足;一个被丢弃的7B模型与一个完整上下文的1.5B模型产生互补错误,其最优选择可恢复至全缓存7B模型87%的准确率差距。基于此,我们提出无训练的推理框架KV-Rescue,通过轻量级全上下文辅助模型与主模型逐步交错生成,形成共享推理轨迹。在线检测器利用熵和可压缩性提前终止不连贯或重复的候选生成。在五个数学基准测试中,使用Qwen2.5-Math 7B和72B模型,在缓存预算B=64时,平均恢复87%的精度损失。解码成本分析表明,防止失控退化使主模型的令牌生成量平均减少43%。

原文摘要 · Abstract (English)

KV-cache eviction caps the memory cost of long reasoning traces but is inherently lossy because the model decodes from a partial view of its history. Under aggressive budgets, this not only lowers accuracy but can also cause runaway degeneration, where the model produces incoherent or repetitive tokens until reaching the length limit. We characterize much of this loss as an information gapf caused by missing context, rather than a capability gap caused by limited model capacity. An evicted 7B model and a full-context 1.5B model make complementary errors, and an oracle choice between their answers recovers 79% of the accuracy gap to the full-KV 7B model. Based on this observation, we propose KV-Rescue, a training-free inference framework that bridges the information gap introduced by KV eviction using a lightweight full-context helper. KV-Rescue interleaves reasoning steps from the two models into a shared trajectory. An online detector uses entropy and compressibility to terminate the generation of incoherent or repetitive base-model candidates early. Across five math benchmarks with Qwen2.5-Math 7B and 72B, KV-Rescue recovers an average of 87% of the accuracy lost to eviction at eviction budget B=64. A decode-cost analysis further shows that preventing runaway degeneration cuts base-model token generation by 43% on average.

长推理缓存优化推理加速大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。