用关联记忆机制解释大模型如何高效存储事实信息。
Understanding Factual Recall in Transformers via Associative Memories
- 用线性与MLP关联记忆实现参数量线性增长的存储能力。
- 单层注意力+MLP结构在参数线性增长时达到100%准确率。
- 揭示模型通过值矩阵或MLP交替存取事实的存储策略。
大型语言模型展现出惊人的事实回忆能力。先前研究发现，训练于事实回忆任务的Transformer模型其信息存储容量与参数量成正比。本文证明，浅层Transformer可通过组合关联记忆实现接近最优的存储容量。我们首先证明线性与MLP关联记忆的存储容量均随参数量线性增长。接着引入一个合成的事实回忆任务，并证明：当自注意力层或MLP层的参数量（至多对数因子）随事实数量线性增长时，仅含单层自注意力与MLP的Transformer可达到100%准确率。模型可在使用值矩阵或MLP作为关联记忆之间灵活权衡。我们进一步分析简化线性注意力模型在该任务上的梯度流轨迹，发现其呈现顺序学习行为。
原文摘要 · Abstract (English)
Large language models have demonstrated an impressive ability to perform factual recall. Prior work has found that transformers trained on factual recall tasks can store information at a rate proportional to their parameter count. In our work, we show that shallow transformers can use a combination of associative memories to obtain such near optimal storage capacity. We begin by proving that the storage capacities of both linear and MLP associative memories scale linearly with parameter count. We next introduce a synthetic factual recall task, and prove that a transformer with a single layer of self-attention followed by an MLP can obtain 100% accuracy on the task whenever either the total number of self-attention parameters or MLP parameters scales (up to log factors) linearly with the number of facts. In particular, the transformer can trade off between using the value matrices or the MLP as an associative memory to store the dataset of facts. We complement these expressivity results with an analysis of the gradient flow trajectory of a simplified linear attention model trained on our factual recall task, where we show that the model exhibits sequential learning behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。