基于用户查询优化缓存融合，让大模型问答更快更准。
QCFuse: Query-Centric Cache Fusion for Efficient RAG Inference
- 用查询语义锚点增强查询表示，全局指导缓存选择
- 减少40%计算开销，保持与现有方法相当的准确率
- 适合需要高效推理的对话系统与知识问答场景
缓存融合通过键值（KV）缓存和选择性重新计算，加速配备检索增强生成（RAG）的大语言模型（LLM）生成过程，降低计算成本并提升效率。然而，现有方法多从局部视角选择令牌，缺乏来自用户查询的全局感知。由于获取上下文感知的查询表示成本高，且高效注意力分析需严格流水线约束，实现全局感知极具挑战。为此，本演示提出QCFuse，一种以用户查询为中心的新型KV缓存融合系统。QCFuse利用语义摘要锚点增强查询表示，选择性地重新计算与查询相关的令牌，并基于最关键Transformer层的注意力分布更新令牌，从而在保持流水线结构高效率的同时，提升准确性。在真实数据集上的评估表明，QCFuse相比当前方法显著提升响应效率达40%，同时维持相当的准确率；此外，在某些场景下，其还表现出注意力去噪效果，带来更高响应准确率，展现出大模型推理优化的巨大潜力。
原文摘要 · Abstract (English)
Cache fusion accelerates generation process of LLMs equipped with RAG through KV caching and selective token recomputation, thereby reducing computational costs and improving efficiency. However, existing methods primarily rely on local perspectives for token selection and lack global awareness from the user query. Utilizing this global awareness is challenging due to the high cost of obtaining context-aware query representations and the strict pipeline constraints required for efficient attention analysis. Thus, this demonstration introduces QCFuse, an innovative KV cache fusion system centered on the user query. QCFuse leverages semantic summary anchors to enhance query representations and selectively recomputes query-related tokens to improve accuracy, updating tokens based on the attention distribution of the most critical Transformer layer to preserve the high efficiency of the pipeline structure. Evaluations on real-world datasets demonstrate that QCFuse significantly improves the response efficiency of LLMs by 40\% while maintaining equivalent accuracy compared to current methods. Additionally, in certain scenarios, QCFuse achieves an attention denoising effect that yields higher response accuracy, demonstrating substantial potential in the optimization of LLM inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。