EpiCache通过分块预填充和话题聚类压缩，让模型在有限内存下保持长对话准确率。
EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environments
- 按话题分段管理缓存，动态丢弃无关内容
- 在3个基准上提升准确率最高达30%，压缩6倍仍接近全缓存效果
- 适合资源受限设备上的长期对话场景
现代大语言模型将上下文长度扩展至数百万标记，实现基于长期对话历史的连贯个性化回复。然而，键值（KV）缓存随对话历史线性增长，导致模型内存占用迅速超出设备限制。现有缓存压缩方法多在处理完整上下文后进行淘汰，造成峰值内存无界增长；且查询依赖的淘汰策略使缓存语义局限于单一查询，难以应对多轮对话。本文提出EpiCache，一种无需训练的KV缓存管理框架，适用于固定内存预算下的长对话问答（LongConvQA）。EpiCache通过分块预填充控制缓存增长，并利用话题式缓存压缩，将对话历史聚类为连贯片段，对每段执行特定淘汰。在三个LongConvQA基准（LongMemEval、Realtalk、LoCoMo）上，EpiCache准确率最高提升30%，在4-6倍压缩下达到近似全缓存精度，延迟和峰值内存分别降低最多2.4倍和3.7倍。
原文摘要 · Abstract (English)
Modern large language models (LLMs) extend context lengths to millions of tokens, enabling coherent, personalized responses grounded in long conversational history. However, the Key-Value (KV) cache grows linearly with the extended dialogue history, causing the model's memory footprint to quickly exceed device limits. While recent KV cache compression methods attempt to reduce memory usage, most apply cache eviction after processing the entire context, incurring unbounded peak memory usage. Additionally, query-dependent eviction narrows the cache semantics to a single query, leading to failure cases in multi-turn conversations. In this paper, we introduce EpiCache, a training-free KV cache management framework for long conversational question answering (LongConvQA) under fixed memory budgets. EpiCache bounds cache growth through block-wise prefill and preserves topic-relevant context via episodic KV compression, which clusters conversation history into coherent episodes and performs episode-specific KV cache eviction. Across three LongConvQA benchmarks (LongMemEval, Realtalk, and LoCoMo), EpiCache improves accuracy by up to 30%, achieves near full-cache accuracy under 4-6x compression, and reduces latency and peak memory by up to 2.4x and 3.7x, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。