M+让大模型记住超过16万字的长期信息，比之前强得多。
M+: Extending MemoryLLM with Scalable Long-Term Memory
- 用可检索的长期记忆机制替代固定压缩，动态调取过往信息。
- 在160k上下文长度下仍能准确保留知识，是之前的8倍以上。
- 适合需要超长记忆的问答、对话和文档理解场景。
为大语言模型配备隐空间记忆已成为扩展上下文窗口的重要方向。然而，远距离历史信息的保留仍是难题。以MemoryLLM（Wang et al., 2024a）为代表的代表性工作将历史信息压缩至所有层的隐藏状态，形成10亿参数的记忆池，虽在16,000词以内有效，但在超过20,000词时知识保持能力显著下降。本文提出M+，基于MemoryLLM构建的增强型记忆模型，引入可训练的检索器与长期记忆机制，在生成过程中动态召回相关历史内容。我们在多个基准上评估M+，涵盖长上下文理解与知识留存任务。实验表明，M+显著优于MemoryLLM及近期强基线，在相似显存开销下，将知识保留上限从不足20,000词提升至超过160,000词。代码已开源：https://github.com/wangyu-ustc/MemoryLLM
原文摘要 · Abstract (English)
Equipping large language models (LLMs) with latent-space memory has attracted increasing attention as they can extend the context window of existing language models. However, retaining information from the distant past remains a challenge. For example, MemoryLLM (Wang et al., 2024a), as a representative work with latent-space memory, compresses past information into hidden states across all layers, forming a memory pool of 1B parameters. While effective for sequence lengths up to 16k tokens, it struggles to retain knowledge beyond 20k tokens. In this work, we address this limitation by introducing M+, a memory-augmented model based on MemoryLLM that significantly enhances long-term information retention. M+ integrates a long-term memory mechanism with a co-trained retriever, dynamically retrieving relevant information during text generation. We evaluate M+ on diverse benchmarks, including long-context understanding and knowledge retention tasks. Experimental results show that M+ significantly outperforms MemoryLLM and recent strong baselines, extending knowledge retention from under 20k to over 160k tokens with similar GPU memory overhead. We open-source our code at https://github.com/wangyu-ustc/MemoryLLM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。