让大模型直接存文本，可查可删，更透明可控。
MeMo: Towards Language Models with Associative Memory Mechanisms
- 用分层联想记忆直接存储文本序列，不依赖学习
- 单层和多层配置均能高效记忆，支持精准编辑
- 适合需要可控记忆、可删除内容的场景
记忆是基于Transformer的大语言模型的一项基本能力，通常通过学习获得。本文提出范式转变：设计一种架构，让模型直接记忆文本，遵循‘记忆先于学习’的原则。我们引入MeMo，一种新型语言建模架构，通过分层联想记忆显式存储标记序列。设计上，MeMo具备透明性，支持模型编辑，包括遗忘特定文本。实验验证了单层与多层配置的记亿能力。
原文摘要 · Abstract (English)
Memorization is a fundamental ability of Transformer-based Large Language Models, achieved through learning. In this paper, we propose a paradigm shift by designing an architecture to memorize text directly, bearing in mind the principle that memorization precedes learning. We introduce MeMo, a novel architecture for language modeling that explicitly memorizes sequences of tokens in layered associative memories. By design, MeMo offers transparency and the possibility of model editing, including forgetting texts. We experimented with the MeMo architecture, showing the memorization power of the one-layer and the multi-layer configurations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。