通过智能排序检索证据，显著降低大模型问答的等待时间。
CacheWeaver: Cache-Aware Evidence Ordering for Efficient Grounded RAG Inference
- 构建前缀树并贪心排序证据，提升缓存复用率。
- 在三个配置下减少20%-33%的首次生成延迟。
- 无需改动模型或检索结果，适合部署优化场景。
检索增强生成（RAG）虽提升事实准确性，却延长提示长度并增加预填充开销。现有服务引擎如vLLM的前缀缓存仅在请求共享相同前缀时有效。但在基于证据的生成中，相邻查询可能以不同顺序检索重叠内容，导致集合重叠无法转化为可复用的前缀重叠。我们提出CacheWeaver，一种轻量级提示层缓存感知证据排序方法。该方法维护最近服务的证据序列前缀树，并通过贪心策略将最具复用性的前缀置于首位，同时保持服务引擎和检索结果不变。在三个vLLM配置下，相比按检索顺序缓存，该方法将中位数首次生成时间（TTFT）降低约20%-33%，且不影响问答质量。贪心策略达到最优排序97.5%的中位数性能增益，表明大部分缓存局部性可通过简单调度层恢复。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) improves factual grounding, but it also lengthens prompts and raises prefill cost. Prefix caching in serving engines such as vLLM reduces this cost only when requests share the same token prefix. In grounded generation, however, adjacent queries may retrieve overlapping evidence in different orders, so set overlap does not become reusable prefix overlap. We present CacheWeaver, a lightweight prompt-layer method for cache-aware evidence ordering. The method keeps a prefix tree over recently served evidence sequences and uses a greedy walk to place the most reusable prefix first, while leaving the serving engine and retrieved evidence set unchanged. Across three vLLM configurations, the method lowers median time-to-first-token (TTFT) by about 20-33 percent relative to retrieval-order prefix caching, without hurting answer quality in our QA tests. The greedy policy reaches 97.5 percent of the median TTFT gain from oracle ordering, indicating that most reusable prefix locality can be recovered by a simple scheduling layer between retrieval and inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。