对比多种缓存淘汰策略,发现LFU在大多数场景下表现最佳。
Which Eviction Policy Should an LLM Cache Use? A Systematic Study Across Workloads, Capacities, and Encoders
- 在三种容量、两种编码器下系统评估7种淘汰策略
- LFU优于其他策略最多仅0.041个百分点,但FIFO和SISO落后超8.5点
- 缓存命中率需结合答案可替换性评估,否则虚高
语义缓存通过查询嵌入的邻近性复用LLM响应,但现有淘汰策略极少在同一协议下系统比较。本文使用CLEVER框架,在三个有序去重查询语料、三种缓存容量和两个编码器下,评估FIFO、LRU、LFU、ARC、GDSF、单遍流式SISO及语义冗余策略。在十八种设置中,无一策略优于LFU超过0.041个百分点。淘汰策略影响显著:在紧约束容量下,FIFO和流式SISO分别落后于LFU达8.67和8.55个百分点。我们通过条件打包结果解释此现象:在精确查找与缺失时插入机制下,新插入项不可能有位于命中半径内的邻近项,导致几何感知淘汰规则难以获取冗余信号。独立审计揭示更严重问题:在MiniLM的中位最近邻阈值下,仅2.1-3.9%的LMSYS和QQP命中被判断为答案可替换,使原始命中率51-60%降至质量调整后1.1-2.2%。跨编码器实验进一步表明,阈值无法在不同嵌入模型间迁移。因此,在该协议下LFU是最佳简单默认选择;部署前应先验证答案有效性,再以精确搜索测试子级策略差异。
原文摘要 · Abstract (English)
Semantic caches reuse an LLM response when the incoming query embedding lies near a cached query, but proposed eviction policies have rarely been compared under one protocol. Using CLEVER, we evaluate FIFO, LRU, LFU, ARC, GDSF, a single-pass streaming adaptation of SISO, and a semantic-redundancy policy across three ordered, deduplicated query corpora, three cache capacities, and two encoders. No evaluated policy improves on LFU by more than 0.041 percentage points in any of the eighteen settings. Replacement is not irrelevant: FIFO and streaming SISO trail LFU by as much as 8.67 and 8.55 points, respectively, at tight capacity. We explain the missing upside with a conditional packing result. Under exact lookup and insert-on-miss, a newly inserted entry cannot have a resident neighbor within the hit radius, so a geometry-aware eviction rule receives little new redundancy signal. A separate audit exposes a larger problem with the evaluated operating point. At MiniLM's median nearest-neighbor threshold, only 2.1-3.9% of sampled LMSYS and QQP hits are judged answer-substitutable, reducing raw hit rates of 51-60% to quality-adjusted rates of 1.1-2.2%. The cross-encoder study further shows that thresholds do not transfer between embedding models. LFU is the strongest simple default in this protocol; deployment decisions should first establish answer validity and then test sub-point policy differences with exact search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。