通过预测生成过程中的语义线索，提前预取信息，显著降低RAG模型延迟。
Predictive Prefetching for Retrieval-Augmented Generation
- 基于生成动态中的语义前兆，预测何时及何内容需预取
- 端到端延迟降低43.5%，首字响应时间减少62.4%
- 适用于复杂多领域场景，适合追求低延迟的生成应用
检索增强生成（RAG）虽能提升大模型的事实准确性，但因同步检索导致显著延迟。现有异步方案依赖启发式协调，且假设解码期间信息需求稳定，这在复杂多领域场景中常失效。本文提出一种先进的异步检索框架，支持与动态信息需求对齐的预测性预取。该框架通过三个组件——检索预测器、上下文监测器和查询生成器——利用生成过程中出现的语义前兆（在不确定性关键前数个词元即显现），明确预测触发检索的时间与内容。多基准测试表明，该方法实现高达43.5%的端到端延迟降低，时间到首个词元改善62.4%，同时保持与同步RAG基线相当的答案质量。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) improves factual grounding in large language models but suffers from substantial latency due to synchronous retrieval. While recent work explores asynchronous retrieval, existing approaches rely on heuristic coordination between retrieval and generation and assume stable information demands during decoding that often break in complex, multi-domain settings. In this paper, we propose an advanced asynchronous retrieval framework that enables predictive prefetching aligned with evolving information needs. The framework explicitly predicts when retrieval should be triggered and what information should be retrieved using three components, a retrieval predictor, a context monitor, and a query generator, by exploiting semantic precursors in generation dynamics that emerge several tokens before uncertainty becomes critical. Experiments on multiple benchmarks demonstrate up to 43.5% end-to-end latency reduction and 62.4% improvement in time-to-first-token, while maintaining answer quality comparable to synchronous RAG baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。