改进分块缓存机制，提升大模型生成质量与推理速度
An experimental study of KV cache reuse strategies in chunk-level caching systems
- 通过实验发现现有分块缓存方法存在根本缺陷
- 新设计融合多种技术，显著提升生成准确率
- 适合需要高效高质生成的部署场景
检索增强生成通过在提示中加入相关检索文本，提升大语言模型的准确性。分块级缓存（CLC）通过预先计算并复用检索块的KV缓存来加速推理。然而，这些缓存忽略了块之间的交叉注意力依赖，可能导致输出质量下降。现有多种方法尝试通过不同技术改善CLC的准确性。本文有两个主要贡献：首先，通过大规模实验评估，揭示现有CLC方法存在根本性局限，制约其准确率或适用范围；其次，观察到现有技术具有互补性，据此提出一种新设计，通过精心组合多种方法，实现更高准确率。
原文摘要 · Abstract (English)
Retrieval-augmented generation improves large language models' accuracy by adding relevant retrieved text to the prompt. Chunk level caching (CLC) accelerates inference by precomputing KV caches for these retrieved chunks and reusing them. However, these caches miss cross-attention dependencies between chunks, which can reduce output quality. Several methods try to improve CLC accuracy using different techniques. We make two main contributions. First, we show that existing CLC approaches have fundamental limitations that limit their accuracy or their applicability. We back this conclusion with an extensive CLC system experimental evaluation. Second, we observe that existing CLC techniques are complementary. We leverage this insight to propose a new CLC design that carefully combines them and achieves better accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。