arXiv:2510.26835cs.DBcs.AI2025-10被引 5

按查询类别动态调整缓存策略,提升大模型服务的缓存效率。

Category-Aware Semantic Caching for Heterogeneous LLM Workloads

  • 根据查询类型自适应调节相似度阈值、过期时间与配额。
  • 将缓存命中率低至3-5%的类别也纳入经济可行范围。
  • 适合高并发、多类型查询的大模型服务系统使用。

大模型服务系统处理异构查询负载,不同类别特性差异显著:代码查询在嵌入空间中聚集紧密,对话类查询分布稀疏;内容陈旧周期从几分钟(股票数据)到数月(代码模式)不等;查询重复模式从幂律分布(代码)到均匀分布(对话)不等,导致缓存命中率呈现长尾分布——高重复类别命中率达40-60%,低重复或高变动类别仅5-15%。向量数据库需排除长尾部分,因远程搜索耗时30ms,需15-20%命中率才能盈亏平衡,导致20-30%生产流量无法缓存。统一缓存策略加剧问题:固定阈值在密集区产生误报,在稀疏区漏掉有效同义句;固定TTL浪费内存或返回过期数据。本文提出类别感知语义缓存,使相似度阈值、TTL和配额随查询类别动态调整。设计混合架构,将内存HNSW搜索与外部文档存储分离,将未命中开销从30ms降至2ms。此降幅使低命中率类别经济可行(盈亏平衡点降至3-5%而非15-20%),实现全工作负载覆盖。基于负载的自适应策略进一步扩展框架,响应下游模型负载,理论预测可使过载模型流量减少9-17%。

原文摘要 · Abstract (English)

LLM serving systems process heterogeneous query workloads where different categories exhibit different characteristics. Code queries cluster densely in embedding space while conversational queries distribute sparsely. Content staleness varies from minutes (stock data) to months (code patterns). Query repetition patterns range from power-law (code) to uniform (conversation), producing long tail cache hit rate distributions: high-repetition categories achieve 40-60% hit rates while low-repetition or volatile categories achieve 5-15% hit rates. Vector databases must exclude the long tail because remote search costs (30ms) require 15--20% hit rates to break even, leaving 20-30% of production traffic uncached. Uniform cache policies compound this problem: fixed thresholds cause false positives in dense spaces and miss valid paraphrases in sparse spaces; fixed TTLs waste memory or serve stale data. This paper presents category-aware semantic caching where similarity thresholds, TTLs, and quotas vary by query category. We present a hybrid architecture separating in-memory HNSW search from external document storage, reducing miss cost from 30ms to 2ms. This reduction makes low-hit-rate categories economically viable (break-even at 3-5% versus 15-20%), enabling cache coverage across the entire workload distribution. Adaptive load-based policies extend this framework to respond to downstream model load, dynamically adjusting thresholds and TTLs to reduce traffic to overloaded models by 9-17% in theoretical projections.

大模型服务缓存优化语义缓存

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。