新缓存系统SISO提升大模型服务效率，显著改善响应速度与资源利用。
Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristics
- 用中心点缓存策略扩大覆盖范围，减少内存占用。
- 在多个数据集上命中率最高提升1.71倍，更稳定达成服务目标。
- 适合追求高吞吐、低延迟的大模型部署场景。
大规模部署大语言模型（LLM）需在严苛的计算与内存约束下满足严格的性能指标。然而，传统缓存策略存在不足：精确匹配和前缀缓存忽略查询语义；现有语义缓存仍受限于传统思路，缺乏根本性突破。为此，我们提出SISO——一种重新定义大模型服务效率的语义缓存系统。SISO引入基于中心点的缓存机制以最大化覆盖率并最小化内存开销，采用局部性感知替换策略保留高价值缓存项，并通过动态阈值调节在不同负载下平衡准确率与延迟。在多种数据集上的实验表明，SISO相比当前最优系统，命中率最高提升1.71倍，且始终更优地达成服务等级目标。
原文摘要 · Abstract (English)
Serving Large Language Models (LLMs) at scale requires meeting strict Service Level Objectives (SLOs) under severe computational and memory constraints. Nevertheless, traditional caching strategies fall short: exact-matching and prefix caches neglect query semantics, while state-of-the-art semantic caches remain confined to traditional intuitions, offering little conceptual departure. Building on this, we present SISO, a semantic caching system that redefines efficiency for LLM serving. SISO introduces centroid-based caching to maximize coverage with minimal memory, locality-aware replacement to preserve high-value entries, and dynamic thresholding to balance accuracy and latency under varying workloads. Across diverse datasets, SISO delivers up to 1.71$\times$ higher hit ratios and consistently stronger SLO attainment compared to state-of-the-art systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。