用语义缓存加速大模型响应，提升效率并降低成本。
From Exact Hits to Close Enough: Semantic Caching for LLM Embeddings
- 基于嵌入相似性重用请求，设计高效缓存策略
- 提出新变体算法，比传统方法提升语义准确率
- 适合追求推理速度与成本优化的系统开发者
大语言模型的快速普及带来了对更快响应和更低计算成本的需求。语义缓存通过利用请求嵌入的语义相似性复用结果来满足这一需求，但打破了传统缓存的假设并引入新挑战。本文研究离线语义缓存策略，证明最优离线策略实现为NP难问题，并提出若干多项式时间启发式方法。同时，提出结合时效性、频率与局部性的在线语义感知缓存策略。在多个数据集上的评估表明，尽管基于频率的策略表现良好，但本文提出的新型变体在语义准确性上进一步提升。研究揭示了当前系统的有效策略，并指明未来创新的巨大空间。所有代码开源。
原文摘要 · Abstract (English)
The rapid adoption of large language models (LLMs) has created demand for faster responses and lower costs. Semantic caching, reusing semantically similar requests via their embeddings, addresses this need but breaks classic cache assumptions and raises new challenges. In this paper, we explore offline policies for semantic caching, proving that implementing an optimal offline policy is NP-hard, and propose several polynomial-time heuristics. We also present online semantic aware cache policies that combine recency, frequency, and locality. Evaluations on diverse datasets show that while frequency based policies are strong baselines, our novel variant improves semantic accuracy. Our findings reveal effective strategies for current systems and highlight substantial headroom for future innovation. All code is open source.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。