让大模型缓存更高效，支持任意前缀下的重复内容复用。
EPIC: Efficient Position-Independent Caching for Serving Large Language Models
- 提出无位置依赖的缓存机制，突破传统缓存必须完全匹配前缀的限制。
- 实测可提升8倍首令牌延迟、7倍吞吐量，精度几乎不变。
- 适合需要频繁调用固定文档的少样本学习与检索生成场景。
大语言模型在众多应用中展现强大能力，但随着请求复杂度上升，服务效率面临挑战。上下文缓存通过复用键值（KV）向量提升性能，这些向量是跨请求重复出现的标记中间表示。然而现有缓存需请求间精确前缀匹配，限制了在少样本学习和检索增强生成等场景中的复用机会，这些场景中不变内容（如文档）虽保持一致，但前缀各异。位置无关缓存（PIC）通过实现不依赖前缀的模块化KV向量复用解决此问题。本文正式定义了PIC，并提出EPIC系统，引入新算法LegoLink，有效缓解每个文档开头的不当‘注意力陷阱’效应，在保持精度的同时仅需极小计算开销。实验表明，相比现有系统，EPIC在时间到首个令牌（TTFT）上最高提升8倍，吞吐量提升7倍，且精度损失可忽略或无。
原文摘要 · Abstract (English)
Large Language Models (LLMs) show great capabilities in a wide range of applications, but serving them efficiently becomes increasingly challenging as requests (prompts) become more complex. Context caching improves serving performance by reusing Key-Value (KV) vectors, the intermediate representations of tokens that are repeated across requests. However, existing context caching requires exact prefix matches across requests, limiting reuse cases in settings such as few-shot learning and retrieval-augmented generation, where immutable content (e.g., documents) remains unchanged across requests but is preceded by varying prefixes. Position-Independent Caching (PIC) addresses this issue by enabling modular reuse of the KV vectors regardless of prefixes. We formalize PIC and advance prior work by introducing EPIC, a serving system incorporating our new LegoLink algorithm, which mitigates the inappropriate "attention sink" effect at every document beginning, to maintain accuracy with minimal computation. Experiments show that EPIC achieves up to 8x improvements in Time-To-First-Token (TTFT) and 7x throughput gains over existing systems, with negligible or no accuracy loss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。