提出新评估指标，解决语义缓存部署效果差的问题
Closing the Operational Gap in Semantic Caching
- 引入P-CHR AUC和ORR两个缓存感知指标
- 发现高PR-AUC模型部署时表现反而最差
- 模型选择本质是阈值效用问题，非排名问题
语义缓存通过为语义相似的查询返回缓存结果来降低大语言模型推理成本。当前标准评估方法使用PR-AUC，仅衡量得分排序能力，忽略在固定阈值下的可用性。我们发现这种不匹配导致部署选择系统性偏差：PR-AUC最高的模型在实际运行中往往表现最差。为此，我们提出精确率-缓存命中率（P-CHR）AUC，衡量不同缓存利用率下的精度；以及运营保留率（ORR），反映离线排名质量在部署时的留存程度。我们将离线与部署性能之间的操作差距分解为可恢复的阈值效用部分和由数据集中正例比例决定的不可还原结构部分。实验表明，阈值效用差距受训练目标而非数据规模影响，仅通过重归一化候选集得分或改变训练目标即可缓解。最终，语义缓存的模型选择本质是阈值效用问题，而非排序问题，正确测量是弥合差距的第一步。
原文摘要 · Abstract (English)
Semantic caching cuts LLM inference costs by serving a cached response to semantically similar queries. Standard practice evaluates these systems using PR-AUC, a metric that only measures how well scores rank and ignores whether they are usable at a fixed threshold. We show this mismatch leads to systematically poor deployment choices, as models with the highest PR-AUC are often the worst in operation. We introduce Precision--Cache Hit Ratio (P-CHR) AUC, a cache-aware metric that measures precision across cache utilization levels, and Operational Retention Rate (ORR), which captures how much offline ranking quality survives at deployment. We decompose the operational gap between offline and deployed quality into a recoverable threshold-utility component and an irreducible structural component fixed by the dataset's positive rate. Our experiments show that the threshold-utility gap is governed by the training objective rather than data scale, and yields only to re-normalizing scores over the candidate pool or changing the training objective. Ultimately, model selection for semantic caching is a threshold-utility problem, not a ranking one, and measuring it is the first step to closing the gap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。