用生成式缓存降低大模型调用延迟和成本，还能应对新问题。
A Generative Caching System for Large Language Models
- 通过合成多个缓存结果，让系统回答从未见过的问题。
- 相比GPTcache，响应速度显著提升，且在成本与延迟间取得更好平衡。
- 适合需要频繁调用大模型的开发者和企业用户。
由于大语言模型（LLMs）通常存在数秒至数分钟不等的高延迟，且多数模型按查询收费，缓存技术可带来显著性能与经济收益。本文提出一种新型缓存系统，不仅降低访问延迟与费用，还引入生成式缓存机制：多个已缓存响应可被合成，以回答未曾出现过的查询。该系统作为信息资源库，支持深度挖掘与分析。同时，我们优化了语义缓存算法，在成本、延迟与回复质量之间实现更优权衡。性能测试表明，本系统较GPTcache有明显提速。
原文摘要 · Abstract (English)
Caching has the potential to be of significant benefit for accessing large language models (LLMs) due to their high latencies which typically range from a small number of seconds to well over a minute. Furthermore, many LLMs charge money for queries; caching thus has a clear monetary benefit. This paper presents a new caching system for improving user experiences with LLMs. In addition to reducing both latencies and monetary costs for accessing LLMs, our system also provides important features that go beyond the performance benefits typically associated with caches. A key feature we provide is generative caching, wherein multiple cached responses can be synthesized to provide answers to queries which have never been seen before. Our generative caches function as repositories of valuable information which can be mined and analyzed. We also improve upon past semantic caching techniques by tailoring the caching algorithms to optimally balance cost and latency reduction with the quality of responses provided. Performance tests indicate that our caches are considerably faster than GPTcache.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。