通过热度感知优化，让RAG推理更快更省资源。
HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placement
- 按知识块访问频率分层，动态调整精度与存储位置。
- 在保持精度几乎不变前提下，首令牌生成速度最高快10.49倍。
- 适合需要低延迟、高并发的智能问答系统使用。
检索增强生成（RAG）通过引入外部知识库提升大语言模型输出准确性，有效缓解幻觉和知识更新延迟问题。然而，外部知识库带来长上下文处理挑战，显著增加内存占用和推理延迟。现有方法通过预计算知识库的键值（KV）并在推理时按需加载来加速。本文针对知识库中不同KV块的访问频率差异，提出一种热度感知的RAG加速系统（HA-RAG）。首先，基于KV块数值分布特性，设计热度感知的混合精度压缩与加载方法，降低磁盘I/O和内存访问开销；其次，提出热度感知的数据放置策略，将高频访问的KV块优先存于高速内存，提升数据访问效率。实验表明，相比TurboRAG，HA-RAG在时间到首个令牌（TTFT）上平均提速2.10倍，最大提速达10.49倍，且精度损失可忽略。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) improves model output accuracy by leveraging external knowledge bases, serving as an effective solution to address hallucination issues and knowledge-update delays in Large Language Models (LLMs). However, the introduction of external knowledge bases presents RAG with challenges in long-context processing, significantly increasing memory consumption and inference latency. Existing research accelerates inference by precomputing Key and Value (KV) of the knowledge base and loading them on-demand during inference. Based on the access frequency of different KV chunks within the external knowledge base, this paper proposes a hotness-aware RAG (HA-RAG) inference optimization system. First, leveraging the numerical distribution of KV chunks, we introduce a hotness-aware mixed-precision compressing and loading method to reduce disk I/O and memory access overhead. Second, we design a hotness-aware data placement strategy that prioritizes storing frequently accessed KV chunks in high-speed memory to improve data access efficiency. Experimental results demonstrate that, compared with TurboRAG, the proposed HA-RAG achieves an average speedup of 2.10x and maximum speedup of 10.49x in Time-To-First-Token (TTFT) with negligible accuracy loss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。