通过自适应压缩与存储分级，显著降低大模型服务延迟。
AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving
- 动态选择压缩算法、率和存储位置，优化缓存命中率。
- 相比静态压缩，延迟降低1.43至2.4倍，或质量提升6%至55%。
- 适合高并发、长上下文的大模型服务场景。
大型语言模型应用常复用已处理的上下文（如聊天历史、文档），导致大量冗余计算。现有服务系统通过存储已处理上下文的键值缓存（KV cache）并在新请求中重用时加载，来缓解此问题。随着应用规模扩大，KV缓存总量激增，需同时使用DRAM和SSD进行存储。然而，先前将缓存存于DRAM和SSD的方案存在加载延迟高问题，因多数缓存命中来自慢速的SSD。为提高DRAM命中率，我们提出一种有损压缩策略：针对每个缓存条目，动态决定压缩算法、压缩率及存储设备，以最大化DRAM命中率并最小化加载延迟，同时保持生成质量。在三个任务上对比多种静态压缩基线，AdaptCache实现1.43至2.4倍延迟节省，或在相同延迟下质量提升6%至55%。
原文摘要 · Abstract (English)
Large language model (LLM) applications often reuse previously processed context, such as chat history and documents, which introduces significant redundant computation. Existing LLM serving systems address such redundant computation by storing the KV caches of processed context and loading the corresponding KV cache when a new request reuses the context. Further, as these LLM applications scale, the total size of KV caches becomes excessively large and requires both DRAM and SSD for full storage. However, prior work that stores KV caches in DRAM and SSD suffers from high loading delays, as most KV cache hits come from SSD, which is slow to load. To increase the KV cache hit rate on DRAM, we identify lossy KV cache compression as a promising approach. We design a lossy compression system that decides the compression algorithm, compression rate and device placement for each KV cache entry to maximise DRAM hits and minimise loading delay without significantly degrading generation quality. Compared to various static compression baselines across three tasks, our system AdaptCache achieves 1.43--2.4 x delay savings at the same quality and 6--55% quality improvements at the same delay.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。