让大模型像人一样睡觉，自动清理旧记忆，解决信息干扰问题。
Learning to Forget: Sleep-Inspired Memory Consolidation for Resolving Proactive Interference in Large Language Models
- 引入睡眠周期机制，动态清理过时的键值缓存。
- 在干扰深度为10时仍保持97%的准确率，远超基线（<18%）。
- 适合需要长期记忆稳定性的对话与推理场景。
大型语言模型（LLMs）面临前向干扰（PI）问题：上下文窗口中的过时信息会干扰当前信息的检索，导致准确率随陈旧关联累积呈对数线性下降，这一瓶颈不受上下文长度影响，且无法通过提示工程缓解。生物大脑通过睡眠依赖的记忆巩固机制——突触下调、选择性重播和目标遗忘——解决类似问题。我们提出SleepGate，一种基于生物启发的框架，在基于Transformer的LLM中引入对键值（KV）缓存的可学习睡眠周期。SleepGate包含三个机制：(1) 冲突感知的时间标签器，检测新条目是否覆盖旧条目；(2) 轻量级遗忘门，训练后选择性淘汰或压缩过时缓存条目；(3) 整合模块，将留存条目合并为紧凑摘要。这些组件在推理过程中以自适应熵触发的微睡眠周期激活。我们设计双阶段训练目标，联合优化清醒期的语言建模与睡眠期的后巩固检索。理论分析表明，SleepGate将干扰范围从O(n)降至O(log n)。在小规模Transformer（4层，793K参数）实验中，睡眠深度为5时达到99.5%的检索准确率，深度为10时仍达97.0%，而五种基线（全缓存、滑动窗口、H2O、StreamingLLM、仅衰减）均低于18%。该框架提供了一种提示工程无法触及的架构级解决方案。
原文摘要 · Abstract (English)
Large language models (LLMs) suffer from proactive interference (PI): outdated information in the context window disrupts retrieval of current values. This interference degrades retrieval accuracy log-linearly as stale associations accumulate, a bottleneck that persists regardless of context length and resists prompt-engineering mitigations. Biological brains resolve an analogous challenge through sleep-dependent memory consolidation: synaptic downscaling, selective replay, and targeted forgetting. We propose SleepGate, a biologically inspired framework that augments transformer-based LLMs with a learned sleep cycle over the key-value (KV) cache. SleepGate introduces three mechanisms: (1) a conflict-aware temporal tagger detecting when new entries supersede old ones; (2) a lightweight forgetting gate trained to selectively evict or compress stale cache entries; and (3) a consolidation module that merges surviving entries into compact summaries. These components activate periodically during inference in sleep micro-cycles, governed by an adaptive entropy-based trigger. We formalize a dual-phase training objective jointly optimizing language modeling during the wake phase and post-consolidation retrieval during the sleep phase. Theoretical analysis shows SleepGate reduces the interference horizon from O(n) to O(log n). In experiments with a small-scale transformer (4 layers, 793K parameters), SleepGate achieves 99.5% retrieval accuracy at PI depth 5 and 97.0% at depth 10, while all five baselines -- full KV cache, sliding window, H2O, StreamingLLM, and decay-only ablation -- remain below 18%. Our framework offers an architecture-level solution that prompt engineering cannot address.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。