通过区域感知机制优化大模型推理缓存,提升记忆效率。
MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference
- 根据上下文区域设定不同优先级与衰减率,动态管理缓存。
- 系统区域内容保留率超90%,远高于基线的54%。
- 适合长序列推理、多轮对话等需高效记忆的场景。
大型语言模型代理在推理过程中会积累异构上下文,包括系统指令、计划、用户输入、检索文档、工具输出和中间推理等,其键值(KV)缓存可能成为主要内存瓶颈。现有淘汰策略通常对所有标记应用相同的基于注意力或时效性的规则,忽略了代理协调器已知的语义结构。我们提出 MemDecay,一种无需训练、区域感知的 KV 缓存淘汰策略。MemDecay 为不同区域的标记分配特定的基础优先级与衰减率,当标记被关注时刷新保留分数,并在固定缓存预算下淘汰得分最低的页面,同时允许关键区域被锁定。我们还提供从测量注意力生命周期校准衰减率的方法。在约 450 和 1,700 标记上下文下使用 Qwen2.5-1.5B 与 3B 进行评估。所有设置中,不同区域的注意力寿命相差一个数量级:系统标记半衰期为 148 至 189 解码步,而草稿区标记仅为 14 至 16 步。锁定机制在所有设置中均保持系统区域事实的全缓存准确率,而基线最高仅能保留 13/24;区域感知保留随上下文增长仍有效,而基于时效的保留则迅速失效。累积注意力保留对未锁定内容表现更优,但消融实验表明当前方案中注意力分数归一化是主要限制。
原文摘要 · Abstract (English)
Large language model (LLM) agents accumulate heterogeneous context, including system instructions, plans, user turns, retrieved documents, tool outputs, and intermediate reasoning, whose key-value (KV) cache can become a major memory bottleneck. Existing eviction policies generally apply the same attention- or recency-based rule to every token, ignoring semantic structure already available to the agent orchestrator. We introduce MemDecay, a training-free, region-aware KV-cache eviction policy. MemDecay assigns tokens region-specific base priorities and decay rates, refreshes retention scores when tokens receive attention, and evicts the lowest-scoring pages under a fixed cache budget while allowing critical regions to be pinned. We also provide a procedure for calibrating decay rates from measured attention lifetimes. We evaluate MemDecay at approximately 450 and 1,700 token contexts using Qwen2.5-1.5B and 3B. Across all settings, attention lifetimes differ by an order of magnitude across regions: system-token half-lives range from 148 to 189 decoding steps, compared with 14 to 16 for scratchpad tokens. Pinning preserves system-region facts at full-cache accuracy in every setting, while no baseline preserves more than 13 of 24. Region-aware retention remains effective as context grows, whereas recency-based retention collapses. Accumulated-attention retention performs better on unpinned content, however, and ablations identify attention-score normalization as the main limitation of the current formulation. These results establish semantic prompt structure as a robust signal for KV-cache management while clarifying how it should be combined with attention-based importance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。