用预训练模型离线构建记忆库,显著提升语言模型容量且开销小。
Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory

- 离线用预训练模型生成记忆,通过精确匹配查找记忆值。
- 2.8B模型下平均得分提升至53.86,超越MoE和原始Engram。
- 适合想低成本扩展模型记忆能力的研究者与工程师。
扩大条件记忆是提升语言模型容量的有前景路径,但现有方法如Engram在预训练中从头学习大内存表,成本高且效果不稳定。本文提出Memory Grafting,利用冻结的梯度模型隐藏状态作为条件n-gram记忆。针对频繁出现的局部n-gram,离线运行梯度模型,将最终标记的隐藏表示存储为记忆值,接收模型通过精确最长后缀匹配查找。检索到的记忆经轻量级投影和门控适配,哈希式Engram回退机制确保未匹配上下文的覆盖率。由于梯度模型仅离线运行,且精确查找复杂度期望为O(1),Memory Grafting以极低训练与推理开销扩展外部隐状态容量。在相同接收模型架构与预训练预算下,实验显示其优于MoE与原始Engram。2.8B规模下,平均基准得分从MoE的51.95、Engram的52.43提升至53.86;0.92B规模下,所有梯度模型变体均优于基线,Qwen3.5-35B-A3B表现最佳。结果表明,预训练模型可作为可复用的外部隐状态构造器,为突破可训练参数限制提供实用路径。
原文摘要 · Abstract (English)
Scaling conditional memory offers a promising way to increase language-model capacity, but existing methods such as Engram learn large memory tables from scratch during pre-training, making memory scaling expensive and sometimes ineffective. We propose Memory Grafting, a conditional memory scaling method that utilizes frozen hidden states from a grafting model as conditional n-gram memory. Given frequent local n-grams, we run the grafting model offline, store final-token hidden representations as memory values, and let the recipient model retrieve them through exact longest-match suffix lookup. Retrieved memories are adapted by lightweight projections and gates, while a hash-based Engram fallback preserves coverage for unmatched contexts. Since the grafting model is only run offline and exact lookup has expected O(1) complexity with respect to memory-bank size, Memory Grafting expands external latent capacity with limited training and inference overhead. Experiments under matched recipient architectures and pre-training budgets show that Memory Grafting improves over both MoE and vanilla Engram baselines. In the 2.8B-scale setting, it improves the average benchmark score from 51.95 for MoE and 52.43 for vanilla Engram to 53.86. In the 0.92B-scale setting, all grafting-model variants improve over the baselines, with Qwen3.5-35B-A3B giving the strongest gains. These results suggest that pretrained models can serve as reusable constructors of external latent memory, providing a practical step toward scaling future language models beyond trainable parameters alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。