通过预计算文档的键值缓存，显著降低RAG系统首词生成延迟。
TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text
- 离线预计算并存储文档的键值缓存，推理时直接调用
- 首词生成时间平均缩短8.6倍，最高速率达9.4倍
- 兼容主流大模型，无需修改模型或系统
当前检索增强生成（RAG）系统在预填充阶段需处理大量检索到的文档块，导致显著的首词生成延迟（TTFT）。为降低计算开销与延迟，本文提出TurboRAG，一种新型RAG系统。其核心是先离线预计算并存储文档的键值（KV）缓存，推理时直接检索已保存的缓存用于预填充，从而消除在线计算KV缓存的开销。此外，我们深入分析了掩码矩阵与位置嵌入机制，并微调预训练语言模型以保持模型精度。该方法适用于大多数现有大语言模型及其应用，无需修改模型或推理系统。在多个RAG基准测试中，TurboRAG相比传统RAG系统将TTFT平均降低8.6倍，最高达9.4倍，同时性能与标准RAG相当。
原文摘要 · Abstract (English)
Current Retrieval-Augmented Generation (RAG) systems concatenate and process numerous retrieved document chunks for prefill which requires a large volume of computation, therefore leading to significant latency in time-to-first-token (TTFT). To reduce the computation overhead as well as TTFT, we introduce TurboRAG, a novel RAG system that redesigns the inference paradigm of the current RAG system by first pre-computing and storing the key-value (KV) caches of documents offline, and then directly retrieving the saved KV cache for prefill. Hence, online computation of KV caches is eliminated during inference. In addition, we provide a number of insights into the mask matrix and positional embedding mechanisms, plus fine-tune a pretrained language model to maintain model accuracy of TurboRAG. Our approach is applicable to most existing large language models and their applications without any requirement in modification of models and inference systems. Experimental results across a suite of RAG benchmarks demonstrate that TurboRAG reduces TTFT by up to 9.4x compared to the conventional RAG systems (on an average of 8.6x), but reserving comparable performance to the standard RAG systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。