解决混合模型前缀缓存效率低的问题，提升长文本处理速度。
Marconi: Prefix Caching for the Era of Hybrid LLMs
- 设计新缓存策略，综合考虑复用概率和计算节省收益。
- 在多种负载下实现最高34.4倍的命中率提升，首字响应时间降低617毫秒。
- 适合需要高效处理长上下文的混合大模型部署场景。
结合注意力层语言建模能力与循环层（如状态空间模型）效率的混合模型，在支持大模型长上下文服务方面日益流行。然而，这类模型特有的就地状态更新机制使得部分序列重叠时无法回滚缓存条目，只能接受完全匹配的缓存命中，导致每个序列产生大量缓存项，复用机会极少。本文提出Marconi，首个支持混合大模型高效前缀缓存的系统。其核心在于创新的准入与淘汰策略，不仅依据访问时间，还基于对不同命中场景的复用概率预测，以及命中带来的计算节省与内存开销之比。在多种工作负载与混合模型上，Marconi相比现有最优前缀缓存系统，令牌命中率最高提升34.4倍（达71.1%），首字响应时间减少617毫秒。
原文摘要 · Abstract (English)
Hybrid models that combine the language modeling capabilities of Attention layers with the efficiency of Recurrent layers (e.g., State Space Models) have gained traction in practically supporting long contexts in Large Language Model serving. Yet, the unique properties of these models complicate the usage of complementary efficiency optimizations such as prefix caching that skip redundant computations across requests. Most notably, their use of in-place state updates for recurrent layers precludes rolling back cache entries for partial sequence overlaps, and instead mandates only exact-match cache hits; the effect is a deluge of (large) cache entries per sequence, most of which yield minimal reuse opportunities. We present Marconi, the first system that supports efficient prefix caching with Hybrid LLMs. Key to Marconi are its novel admission and eviction policies that more judiciously assess potential cache entries based not only on recency, but also on (1) forecasts of their reuse likelihood across a taxonomy of different hit scenarios, and (2) the compute savings that hits deliver relative to memory footprints. Across diverse workloads and Hybrid models, Marconi achieves up to 34.4$\times$ higher token hit rates (71.1% or 617 ms lower TTFT) compared to state-of-the-art prefix caching systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。