Memory Mosaics在大规模模型和真实数据上仍保持优异的推理与学习能力。
Memory Mosaics at scale
- 采用新型架构（Memory Mosaics v2），通过网络化关联记忆实现高效知识存储。
- 在1万亿令牌训练下，新任务推理性能显著优于同规模Transformer模型。
- 适合需要快速适应新任务的场景，如增量学习与动态知识更新。
Memory Mosaics [Zhang et al., 2025] 是一种基于关联记忆的网络结构，在中等规模模型（GPT-2级别）和合成小数据集上已展现出色的组合性与上下文学习能力。本文证明，这些优势在扩展至大语言模型规模（Llama-8B级别）和真实数据集时依然成立。为此，我们构建了100亿参数的Memory Mosaics v2，基于一万亿令牌进行训练，并从三个维度评估其能力：训练知识存储、新知识存储及上下文学习。实验表明，Memory Mosaics v2 在训练知识学习方面与Transformer持平，但在推理阶段执行新任务的能力显著更优。即使使用八万亿令牌训练的Transformer，其表现仍不及仅用一万亿令牌训练的Memory Mosaics v2。这一提升无法通过单纯增加训练数据获得。
原文摘要 · Abstract (English)
Memory Mosaics [Zhang et al., 2025], networks of associative memories, have demonstrated appealing compositional and in-context learning capabilities on medium-scale networks (GPT-2 scale) and synthetic small datasets. This work shows that these favorable properties remain when we scale memory mosaics to large language model sizes (llama-8B scale) and real-world datasets. To this end, we scale memory mosaics to 10B size, we train them on one trillion tokens, we introduce a couple architectural modifications ("Memory Mosaics v2"), we assess their capabilities across three evaluation dimensions: training-knowledge storage, new-knowledge storage, and in-context learning. Throughout the evaluation, memory mosaics v2 match transformers on the learning of training knowledge (first dimension) and significantly outperforms transformers on carrying out new tasks at inference time (second and third dimensions). These improvements cannot be easily replicated by simply increasing the training data for transformers. A memory mosaics v2 trained on one trillion tokens still perform better on these tasks than a transformer trained on eight trillion tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。