用多向量检索和可学习分段提升大模型缓存命中率
MVR-cache: Optimizing Semantic Caching via Multi-Vector Retrieval and Learned Prompt Segmentation
- 通过可学习分段将提示拆解，实现细粒度相似度匹配
- 在多个任务上最高提升37%缓存命中率，且保证结果正确性
- 适合需要降低大模型推理成本与延迟的系统开发者
为降低大模型的成本与延迟，语义缓存系统必须准确判断新提示是否与缓存项匹配。现有方法常依赖简单相似度度量，效果受限。本文提出MVR-cache，通过引入多向量检索（MVR）显著提升检索准确性。该系统基于可学习分段模型，智能分割提示，实现基于MaxSim的细粒度相似性比较。模型训练目标由严格理论分析推导得出，优化该目标可直接最大化缓存命中率，同时满足严格的正确性约束。针对由此产生的不可微组合优化问题，采用基于强化学习的训练策略，以理论目标作为奖励信号。在多种任务的基准测试中，MVR-cache相比现有最优方法，缓存命中率最高提升37%，且保持相同正确性保障。代码已开源：https://github.com/PKU-SDS-lab/MVR-Cache
原文摘要 · Abstract (English)
To reduce LLM costs and latency, semantic caching systems must accurately identify when a new prompt matches a cached one. Current methods often rely on simplistic similarity measures, which limit their effectiveness. We introduce MVR-cache, a novel semantic caching approach that significantly improves retrieval accuracy by integrating Multi-Vector Retrieval (MVR). MVR-cache is built upon a learnable segmentation model that intelligently splits prompts, enabling fine-grained similarity comparisons via MaxSim. We derive the model's training objective from a rigorous theoretical analysis. This can ensure that optimizing this objective directly maximizes cache hits under strict correctness constraints. To solve the resulting non-differentiable combinatorial optimization problem, we leverage a reinforcement learning-based training strategy with the theoretically grounded objectives as the reward. Experimental results on established benchmarks across diverse tasks confirm that in comparison to the state-of-the-art, MVR-cache consistently increases the cache hit rates by up to 37% while maintaining the same correctness guarantees. MVR-cache is available at https://github.com/PKU-SDS-lab/MVR-Cache
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。