给解码器模型加编码器，实现无位置依赖的缓存加速推理
You Need an Encoder for Native Position-Independent Caching
- 在解码器模型中引入编码器，显式支持无位置约束的键值缓存
- 使首次生成时间减少51%-94%，吞吐量提升3倍，精度接近原模型
- 适用于多种解码器模型，可无缝集成到现有推理系统
大型语言模型（LLM）的键值（KV）缓存基于前缀，导致在任意顺序检索上下文时效率低下。无位置依赖缓存（PIC）被提出以实现无需位置约束的KV重用，但现有方法常导致显著精度下降，限制了实际应用。为此，我们通过重新引入编码器到主流的仅解码器LLM中，并显式训练其支持PIC，提出了原生PIC方案。我们进一步开发了COMB——一种面向PIC的缓存系统，可无缝集成至现有推理框架。实验表明，COMB将首次生成时间（TTFT）降低51%-94%，吞吐量提升3倍，且精度相当。此外，使用DeepSeek-V2-Lite-Chat的实验验证了COMB对其他类型解码器模型的适用性。代码已公开于https://github.com/shijuzhao/Comb。
原文摘要 · Abstract (English)
The Key-Value (KV) cache of Large Language Models (LLMs) is prefix-based, making it highly inefficient for processing contexts retrieved in arbitrary order. Position-Independent Caching (PIC) has been proposed to enable KV reuse without positional constraints; however, existing approaches often incur substantial accuracy degradation, limiting their practical adoption. To address this issue, we propose native PIC by reintroducing the encoder to prevalent decoder-only LLMs and explicitly training it to support PIC. We further develop COMB, a PIC-aware caching system that integrates seamlessly with existing inference frameworks. Experimental results show that COMB reduces Time-to-First-Token (TTFT) by 51-94% and increases throughput by 3$\times$ with comparable accuracy. Furthermore, the quality improvement when using DeepSeek-V2-Lite-Chat demonstrates the applicability of COMB to other types of decoder-only LLMs. Our code is available at https://github.com/shijuzhao/Comb.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。