arXiv:2603.13289cs.LGcs.AI2026-03被引 2

通过复用前序代理的解码缓存,显著加速多代理大模型协作。

RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse

  • 直接复用前序代理的解码阶段KV缓存,用于后续预填充阶段。
  • 在多种任务上实现超80%的缓存复用率,最快提速4.7倍。
  • 无需训练,精度几乎无损,适合高效多代理系统部署。

随着AI任务复杂度提升,大模型系统正从单体模型转向多代理架构。然而,这种协作模式引入了关键瓶颈:前序代理生成的共享内容重复进行预填充计算,导致KV缓存内存占用高、首令牌延迟(TTFT)增加。现有缓存方法或无法保持代理输出的准确性,或因严格约束导致复用率低。本文提出RelayCaching,一种无需训练的推理方法,直接在后续预填充阶段复用前序代理的解码阶段KV缓存。核心洞察是:相同内容的KV缓存在不同阶段高度一致,而由前缀引起的偏差仅存在于局部层和有限位置。通过选择性重计算这些位置的缓存,可维持模型精度且开销极小,实现优于现有方法的准确率-效率平衡。在数学推理、通用知识和代码生成等多样化协作任务上的实验表明,RelayCaching实现超过80%的缓存复用率,相比标准流水线将TTFT降低最高达4.7倍,且精度损失可忽略。

原文摘要 · Abstract (English)

The increasing complexity of AI tasks has shifted the paradigm from monolithic models toward multi-agent large language model (LLM) systems. However, these collaborative architectures introduce a critical bottleneck: redundant prefill computation for shared content generated by previous agents, which significantly increases KV cache memory usage and time-to-first-token (TTFT). While various KV cache methods have been proposed to mitigate prefill redundancy, they either fail to maintain accuracy on agent-generated outputs or exhibit low reuse rates due to rigid constraints. We present RelayCaching, a training-free inference method that directly reuses decoding phase KV caches from previous agents in subsequent prefill phases. Our key insight is that KV caches for identical content are highly consistent across phases, while prefix-induced deviations are sparse and localized within a limited range of layers and token positions. By selectively recomputing KV caches at these positions, RelayCaching preserves model accuracy with minimal overhead, yielding a superior accuracy-efficiency trade-off over existing methods. Experiments on diverse collaborative LLM tasks spanning mathematical reasoning, general knowledge, and code generation demonstrate that RelayCaching achieves over 80% KV cache reuse, reduces TTFT by up to $4.7\times$ compared to the standard pipeline, all with negligible accuracy degradation.

大模型协作缓存复用推理加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。