让不同大模型的缓存能互相通用,提升多模型协作效率。
Mixture-of-Translators: Translating KV Caches Across Heterogeneous Large Language Models

- 用多个翻译模块捕捉不同模型间的缓存映射关系。
- 在多种模型间保持问答准确率,最高达51.0%闭集准确率。
- 适合需要跨模型共享记忆的长文本生成与多智能体系统。
异构大语言模型系统依赖共享上下文、检索证据和多智能体对话历史,但其内部键值(KV)缓存仍为模型特有,无法跨架构复用。导致每个模型需重复预填充或存储相同上下文,限制了多模型推理与长文本生成的可扩展性。本文提出混合翻译器(Mixture-of-Translators, MoT),将源模型的上下文缓存映射至目标模型的缓存空间。不同于依赖单一投影路径或全局共享隐空间的方法,MoT采用多个翻译模块以捕捉多样化的源-目标映射关系。为降低残余翻译误差,引入上下文校正损失,使重放的目标轨迹与原生目标轨迹对齐。我们揭示了缓存翻译中的两种竞争性失败模式:早期注入导致的传播偏移和晚期注入导致的最终状态偏移,并通过翻译器混合与目标侧校正加以解决。在Qwen2.5、GPT-2和OPT模型间进行同质与异质翻译实验,MoT保持下游问答性能,其中Qwen2.5-7B级翻译达到51.0%平均闭集问答准确率和0.43平均抽取式问答F1。实际案例显示,MoT支持高质量记忆复用,多智能体推理中保留96.3%的直接上下文质量,实现异构大模型间可扩展的KV缓存复用。
原文摘要 · Abstract (English)
Heterogeneous Large Language Model (LLM) systems increasingly rely on shared contexts, retrieved evidence, and multi-agent dialogue histories, yet their internal key-value (KV) caches remain model-specific and cannot be reused across architectures. Consequently, each model must repeatedly prefill or store caches for the same context, limiting the scalability of multi-model reasoning and long-context generation. We propose Mixture-of-Translators(MoT), a cache translation framework that maps context KV caches from a source LLM into the cache space of a target LLM. Unlike prior approaches that depend on a single projection path or global shared latent space, MoT uses multiple translator modules to capture diverse source--target mappings. To further reduce residual translation error, we introduce a Context Correction Loss that aligns the replayed target trajectory with the native target trajectory. We reveal two competing failure modes in cache translation: propagated translation shift from early injection and last-state shift from late injection. MoT addresses them through translator mixtures and target-side correction. Across homogeneous and heterogeneous translations among Qwen2.5, GPT-2, and OPT models, MoT preserves downstream QA performance, including Qwen2.5-7B-scale translation with 51.0% average closed-set QA accuracy and 0.43 average extractive QA F1. In practical case studies, MoT enables quality-preserving memory reuse for multi-agent reasoning and retains 96.3% of direct-context quality in long-context cache-augmented generation, demonstrating scalable KV cache reuse across heterogeneous LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。