用语义缓存复用上下文摘要，让大模型问答更省计算。
Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models
- 将上下文摘要进行语义缓存，支持相似问题复用
- 减少50%-60%冗余计算，答案准确率接近全量处理
- 适合需要低延迟的实时问答系统使用
大型语言模型（LLMs）正被广泛部署于边缘与云端平台，用于实时问答和检索增强生成。然而，在分布式系统中处理长上下文会带来高计算开销、内存占用和网络带宽消耗。本文提出一种新型语义缓存方法，用于存储和重用中间上下文摘要，实现基于LLM的问答流程中相似查询的信息高效复用。在NaturalQuestions、TriviaQA及一个合成的ArXiv数据集上验证，该方法可减少50%-60%的冗余计算，同时保持与完整文档处理相当的答案准确性。该方案在计算成本与响应质量之间取得良好平衡，对实时AI助手至关重要。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly deployed across edge and cloud platforms for real-time question-answering and retrieval-augmented generation. However, processing lengthy contexts in distributed systems incurs high computational overhead, memory usage, and network bandwidth. This paper introduces a novel semantic caching approach for storing and reusing intermediate contextual summaries, enabling efficient information reuse across similar queries in LLM-based QA workflows. Our method reduces redundant computations by up to 50-60% while maintaining answer accuracy comparable to full document processing, as demonstrated on NaturalQuestions, TriviaQA, and a synthetic ArXiv dataset. This approach balances computational cost and response quality, critical for real-time AI assistants.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。