让大模型独立扩展记忆能力，用更少参数提升性能。
Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory
- 设计可独立扩展的记忆模块，分离记忆与推理参数。
- 6.9B模型配1.7B记忆，比12B基线少39%参数但得分更高。
- 适合追求高效性能的模型部署者和记忆增强研究者。
Decoder-only语言模型将长期记忆与推理耦合在单一参数集内，难以独立扩展记忆容量。本文提出Memory Decoder at Scale，将记忆模型规模扩展至6.9B参数，并在300B tokens上预训练。在此数据量下，标准Faiss索引与检索开销过大，难以支撑。为此，我们设计分布式Faiss管道与稀疏批加载kNN分布方案，解决瓶颈。实验表明，将更多参数分配给记忆模块，相比单纯扩大基础模型，能带来更优的参数-性能权衡。在17个基准测试中，6.9B通用记忆搭配Pythia-410M，平均分从29.86提升至37.34，超越参数更大的Pythia-12B（37.24），且仅需39%总参数。对于Qwen3 Base模型（0.6B至14B），1.7B领域记忆在各尺度均使三域平均分提升超9分。结果证明，独立扩展预训练记忆是提升语言模型性能的更高效路径。
原文摘要 · Abstract (English)
Decoder-only language models entangle long-term memory and reasoning in a single parameter set, making it difficult to scale memory capacity independently. Memory Decoder introduces a parametric long-term memory module but only studies it at a relatively small scale. In this work, we present Memory Decoder at Scale, scaling memory models up to 6.9B parameters and pretraining them on 300B tokens. At this data scale, the combined cost of indexing and search makes a standard Faiss pipeline infeasible. We address this bottleneck with a distributed pipeline for Faiss indexing and retrieval, together with sparse, batch-wise loading of kNN distributions. Across model scales, we find that allocating more parameters to memory yields a better parameter-performance tradeoff than scaling the base model alone. On 17 benchmarks, pairing a 6.9B general memory with Pythia-410M raises its average score from 29.86 to 37.34, surpassing Pythia-12B (37.24) with 39% fewer total parameters. For Qwen3 Base models ranging from 0.6B to 14B, 1.7B domain memories improve the average score across the three domains by more than 9 points at every scale. Overall, our results demonstrate that independently scaling pretrained memory offers a more parameter efficient path to improving language model performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。