让冻结的GPT-2模型具备持续记忆能力,只用小适配器就能记住信息。
Trained Persistent Memory for Frozen Decoder-Only LLMs
- 用自注意力机制注入记忆,仅训练一个小适配器θ_mem
- 在1倍容量下三类方法记忆保留率达7-18%,知识增益ΔK达7-10
- 适用于希望给大模型加记忆功能的研究者或开发者
Decoder-only语言模型是无状态的:每轮前向传播后隐藏表示会被丢弃,无法跨会话保持。Jeong (2026a) 表明,训练过的记忆适配器可为冻结的编码器-解码器主干提供持续的潜在空间记忆,基于 Jeong (2026b,c) 的横向记忆框架。本文探讨该原理是否适用于无交叉注意力路径的解码器仅架构,记忆必须通过自注意力单独进入。我们对六种方法——前缀、并行交叉注意力、KV扩展、海布记忆、上下文门控分支和基于槽的稀疏写入——进行了适配,仅在冻结的 GPT-2 上训练一个小型适配器 $θ_{mem}$。写入规则统一;仅读取注入方式从解码器交叉注意力改为自注意力的 KV 前缀或并行分支。在 LoCoMo 数据集上,观察到显著的归纳偏置二分:在 $1\times$ 容量下,三种具有强结构先验的方法(交叉注意力(M.2)、海布记忆(M.4)、槽写入(M.6))分别达到 7-18% 的记忆保留率与 7-10 的知识增益 ΔK;其余三种方法表现差(< 0.4%)。在 $10\times$ 容量下,六种方法全部收敛,表明差异源于架构而非本质限制。结合 Jeong (2026a) 的编码器-解码器结果及 Jeong (2026b,c) 的脑启发模块,这些发现确立了持续潜在空间记忆作为横跨主流 Transformer 架构的通用范式。
原文摘要 · Abstract (English)
Decoder-only language models are stateless: hidden representations are discarded after every forward pass and nothing persists across sessions. Jeong (2026a) showed that trained memory adapters give a frozen encoder-decoder backbone persistent latent-space memory, building on the lateral-memory framework of Jeong (2026b,c). Here we ask whether the same principle transfers to the decoder-only setting, where no cross-attention pathway exists and memory must enter through self-attention alone. We adapt six methods -- prefix, parallel cross-attention, KV extension, Hebbian memory, context-gated branch, and slot-based sparse write -- to a frozen GPT-2, training only a small adapter $θ_{mem}$. The write rule is shared; only the read injection changes from decoder cross-attention to self-attention KV prefix or parallel branch. On LoCoMo we find a striking inductive-bias dichotomy: at $1\times$ capacity, three methods with strong architectural priors -- cross-attention (M.2), Hebbian (M.4), and slot write (M.6) -- achieve retained-memory scores of $7-18\%$ and knowledge gains $ΔK$ of $7-10$, while the other three fail ($< 0.4\%$). At $10\times$ capacity all six converge, showing the gap is architectural, not fundamental. Together with the encoder-decoder results of Jeong (2026a) and the brain-inspired modules of Jeong (2026b,c), these findings establish persistent latent-space memory as a general paradigm spanning major transformer families.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。