提出无需训练的内存整理方法，提升大模型长期记忆稳定性。
MemDefrag: Latent Memory Defragmentation for Large Language Models
- 利用中间层注意力密度识别关键记忆片段，实现无监督定位。
- 50次更新后知识保留率43.0%，远超原有方法的17.4%~17.6%。
- 适合作为通用框架集成至各类大模型与潜在记忆系统中。
潜在记忆通过每层隐藏状态存储过往知识片段，成为大语言模型长时记忆的有前景范式（如MemoryLLM和M+）。然而，该范式在内存更新时因位置编码错位且缺乏区分目标与无关片段的追踪机制，导致性能显著下降。我们探测了各层注意力密度，发现少数中间层始终对目标片段集中最高密度，揭示了内在追踪信号。基于此，提出无需训练、模型无关的MemDefrag框架：(1) 利用中间层追踪信号对记忆进行排序、重排和过滤；(2) 当容量超限时，采用信息量引导的按比例遗忘机制。实验表明，MemDefrag在知识保留（50次更新后达43.0%）和长上下文任务上显著优于MemoryLLM和M+，并在多种大模型及潜在记忆变体间具有良好泛化性。代码已开源于github.com/ryehr/MemDefrag。
原文摘要 · Abstract (English)
Latent memory, which stores past knowledge fragments as per-layer hidden states, has emerged as a promising paradigm (e.g., MemoryLLM and M+) for long-term memory in large language models (LLMs). However, the paradigm suffers from significant performance degradation during memory updates, due to positional encoding misalignment and the absence of any tracing mechanism to distinguish target memory fragments from irrelevant ones. To discover such a tracing mechanism, we probe the layer-wise attention density over stored memory fragments, and find that a small set of middle transformer layers consistently concentrates the highest density on the target fragment - exposing an inherent tracing signal. In light of this, we propose MemDefrag, a training-free and model-agnostic framework that (1) uses a middle-layer tracing signal to conduct memory defragmentation (rank, reorder, and filter memories), and (2) applies an informativeness-guided proportional forgetting mechanism once capacity is exceeded. Experiments show that MemDefrag substantially outperforms MemoryLLM and M+ on knowledge retention (e.g., 43.0% vs. 17.4%/17.6% after 50 memory updates) and long-context benchmarks, and generalizes well across various LLMs and latent-memory variants. The code is available at github.com/ryehr/MemDefrag.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。