通过重用键值缓存提升检索增强生成的效率与质量
HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse
- 利用文档侧键值缓存复用技术优化重排器推理
- 使用解码器仅模型实现2-3倍吞吐提升
- 适合需要高并发低延迟的RAG系统部署
检索增强生成（RAG）通过将外部知识融入大语言模型的生成过程，显著提升了模型性能。其中重排器是关键组件，用于从候选文档中筛选最相关的内容，从而提高生成结果的质量。然而，重排器引入了计算开销，限制了系统的吞吐量和响应速度。为此，本文提出HyperRAG，通过重用文档侧的键值缓存（KV-cache），在保证生成质量的前提下，显著提升RAG管道的效率。为充分发挥缓存复用优势，HyperRAG还集成了一系列系统级优化策略，增强可扩展性与运行效率。实验表明，采用解码器仅架构的重排器时，HyperRAG实现了2-3倍的吞吐量提升，同时在下游任务上表现优于传统RAG服务。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for enhancing the performance of large language models (LLMs) by integrating external knowledge into the generation process. A key component of RAG pipelines is the reranker, which selects the most relevant documents from a pool of retrieved candidates and significantly improves the quality of the generated responses. While rerankers refine the selection of retrieved documents in RAG pipelines, they introduce computational challenges that hinder high throughput and low latency. To address this problem, we propose HyperRAG, a system that optimizes the trade-off between quality and efficiency in RAG pipelines by leveraging KV-cache reuse for efficient reranker inference. By reusing document-side KV-cache, HyperRAG achieves both high-quality generation and system-level efficiency. To fully realize the benefits of KV-cache reuse, HyperRAG incorporates a range of system-level optimizations designed to enhance efficiency and scalability. Experiments show that HyperRAG achieves a 2 - 3 throughput improvement with decoder-only rerankers while also delivering higher downstream performance compared with traditional RAG service.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。