用语义缓存减少大模型调用，降本提速。
GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Caching
- 将用户问题嵌入向量存入内存缓存，匹配相似问法。
- 减少68.8%的API调用，命中率61.6%~68.8%。
- 适合高频重复问答场景，如客服系统。
大型语言模型（如GPT）在多种应用中实现了类人文本的理解与生成，但频繁调用API带来的高算力和成本开销成为瓶颈，尤其在客服聊天机器人等重复性查询场景中。本文提出GPT语义缓存，将用户查询的语义嵌入向量存储于内存（Redis），通过识别语义相似问题，直接返回预生成回复，避免重复调用LLM。实验表明，该方法在多个查询类别中可减少高达68.8%的API调用，缓存命中率达61.6%至68.8%，且正向命中率超过97%，证明缓存响应的可靠性。该技术显著降低运营成本并提升响应速度，有效优化了基于LLM的应用性能。
原文摘要 · Abstract (English)
Large Language Models (LLMs), such as GPT, have revolutionized artificial intelligence by enabling nuanced understanding and generation of human-like text across a wide range of applications. However, the high computational and financial costs associated with frequent API calls to these models present a substantial bottleneck, especially for applications like customer service chatbots that handle repetitive queries. In this paper, we introduce GPT Semantic Cache, a method that leverages semantic caching of query embeddings in in-memory storage (Redis). By storing embeddings of user queries, our approach efficiently identifies semantically similar questions, allowing for the retrieval of pre-generated responses without redundant API calls to the LLM. This technique achieves a notable reduction in operational costs while significantly enhancing response times, making it a robust solution for optimizing LLM-powered applications. Our experiments demonstrate that GPT Semantic Cache reduces API calls by up to 68.8% across various query categories, with cache hit rates ranging from 61.6% to 68.8%. Additionally, the system achieves high accuracy, with positive hit rates exceeding 97%, confirming the reliability of cached responses. This technique not only reduces operational costs, but also improves response times, enhancing the efficiency of LLM-powered applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。