让不同模型共享计算缓存,显著降低重复推理开销。
A Universal Context-Reuse Layer for Cross-Model KV Sharing

- 提出跨模型KV缓存复用机制,实现异构模型间上下文迁移。
- 在跨家族场景下,推理延迟从899毫秒降至138毫秒,精度损失仅1.7%。
- 适用于多模型协作、异构推理等复杂系统,提升整体效率。
现代大语言模型服务系统越来越多地处理重复或共享的上下文,但每个模型仍会独立进行prefill计算,即使其他模型已处理过相同输入。现有KV缓存复用机制虽能减少单模型内的冗余计算,但通常假设缓存生产者与消费者为同一模型。本文研究跨模型KV共享,将源模型生成的KV状态转换为可被不同目标模型使用的表示,包括规模、架构、注意力配置、分词器及模型家族均不同的模型。我们在同族与跨族设置下评估该方法。在Qwen2.5-7B→Qwen2.5-1.5B场景中,翻译后的KV状态使LongBench2准确率从27.59%提升至34.48%,优于原生1.5B基线6.89个百分点,同时降低交接成本。在跨族设置Qwen2.5-1.5B→Gemma-2-2B中,4K上下文长度下目标端prefill成本最多降低67.05%,解码困惑度接近原生基线。在更异构的Llama3.1-70B→Qwen2.5-7B设置中,跨族交接实现44.0%准确率(原生为45.7%),测量延迟由899ms降至138ms。结果初步表明,KV状态可作为可迁移的计算表征,而非严格模型本地缓存,推动‘上下文移动性’作为系统抽象,用于减少异构LLM与多智能体推理工作流中的冗余prefill。
原文摘要 · Abstract (English)
Modern large language model (LLM) serving systems increasingly operate over repeated or shared context, yet each model typically performs its own prefill computation even when another model has already processed the same input. Existing KV-cache reuse mechanisms substantially reduce redundant computation within a single model, but generally assume that the producer and consumer of a cache are identical. We study \emph{cross-model KV sharing}, which translates the KV state produced by a source model into a representation that can be consumed by a different target model, including models that differ in scale, architecture, attention configuration, tokenizer, and model family. We evaluate the approach in both within-family and cross-family settings. For Qwen2.5-7B $\rightarrow$ Qwen2.5-1.5B, translated KV states improve LongBench2 accuracy from 27.59\% to 34.48\%, a gain of 6.89 percentage points over the native 1.5B baseline, while reducing handoff cost relative to native target prefill. For the cross-family Qwen2.5-1.5B $\rightarrow$ Gemma-2-2B setting, KV handoff reduces target-side prefill cost by up to 67.05\% at 4K context length while maintaining decoding perplexity close to native-model baselines. In a more heterogeneous Llama3.1-70B $\rightarrow$ Qwen2.5-7B setting, cross-family handoff achieves 44.0\% accuracy compared with 45.7\% for native Qwen2.5-7B inference, while reducing measured latency from 899ms to 138ms. These results provide initial evidence that KV states can serve as transferable computational representations rather than strictly model-local caches, and motivate \emph{context mobility} as a systems abstraction for reducing redundant prefill across heterogeneous LLM and multi-agent inference workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。