通过前瞻检索提升RAG推理速度，降低显存占用。
TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval
- 前瞻检索预加载数据，与大模型生成并行进行。
- 单查询延迟降低1.53倍，批量吞吐提升1.83倍。
- 适合资源受限场景下的RAG高效部署。
检索增强生成（RAG）通过引入外部数据源提升大语言模型的事实准确性和领域覆盖。现代RAG系统依赖大规模数据存储，带来显著的系统挑战：在GPU显存有限的情况下，实现高吞吐和低延迟困难。为解决此问题，我们提出TeleRAG，一种高效的推理系统，可显著降低延迟、提升吞吐，且对显存需求极低。其核心创新是前瞻检索——一种预测所需数据并将其从CPU提前传输至GPU的预取机制，与LLM生成并行执行。此外，TeleRAG采用预取调度器和缓存感知调度器，支持多GPU高效推理，开销极小。评估表明，TeleRAG在单查询场景下平均端到端延迟减少1.53倍，在批量处理中平均吞吐提升1.83倍，且具备良好可扩展性。这证实了TeleRAG在加快RAG应用部署、降低内存消耗方面的实际价值。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) extends large language models (LLMs) with external data sources to enhance factual correctness and domain coverage. Modern RAG pipelines rely on large datastores, creating a significant system challenge: achieving high throughput and low latency is difficult, especially when GPU memory is limited. To address these challenges, we propose TeleRAG, an efficient inference system that reduces latency and improves throughput with minimal GPU memory requirements. The core innovation of TeleRAG is lookahead retrieval, a prefetching mechanism that predicts required data and transfers them from CPU to GPU in parallel with LLM generation. In addition, TeleRAG adopts a prefetching scheduler and a cache-aware scheduler to support efficient multi-GPU inference with minimal overhead. Evaluations show TeleRAG achieves up to a 1.53x average end-to-end latency reduction (single-query) and 1.83x higher average throughput (batched), as well as good scalability in throughput. This confirms the practical utility of TeleRAG for faster and more memory-efficient deployments of RAG applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。