用联想记忆视角重新解读Transformer，揭示其记忆与更新机制。
Understanding Transformer from the Perspective of Associative Memory
- 将注意力机制类比为联想记忆，从核方法角度解释Softmax注意力有效性。
- 提出检索信噪比衡量记忆容量，发现FFN也具联想记忆特性。
- 统一框架解析不同变体的更新机制，启发模型设计创新。
本文从联想记忆这一经典心理学概念出发，重新审视Transformer架构。从简单的线性注意力开始，探讨两个核心维度：记忆容量——模型能记住多少内容及记忆质量如何？我们引入检索信噪比（retrieval SNR）进行量化，并通过核视角数学揭示Softmax注意力为何高效；同时指出前馈网络（FFN）本质上也是一种联想记忆，为设计优化提供新思路。记忆更新方面，我们构建统一框架，解析不同Transformer变体（如DeltaNet和Softmax注意力）如何更新知识库。由此引发两个深刻问题：1. Transformers在表达能力上是否存在根本限制？能否突破？2. 若上下文无限，模型是否会变得无限智能？本研究旨在揭开Transformer的神秘面纱，澄清现有设计逻辑，激发全新创新路径。
原文摘要 · Abstract (English)
In this paper, we share our reflections and insights on understanding Transformer architectures through the lens of associative memory--a classic psychological concept inspired by human cognition. We start with the basics of associative memory (think simple linear attention) and then dive into two dimensions: Memory Capacity: How much can a Transformer really remember, and how well? We introduce retrieval SNR to measure this and use a kernel perspective to mathematically reveal why Softmax Attention is so effective. We also show how FFNs can be seen as a type of associative memory, leading to insights on their design and potential improvements. Memory Update: How do these memories learn and evolve? We present a unified framework for understanding how different Transformer variants (like DeltaNet and Softmax Attention) update their "knowledge base". This leads us to tackle two provocative questions: 1. Are Transformers fundamentally limited in what they can express, and can we break these barriers? 2. If a Transformer had infinite context, would it become infinitely intelligent? We want to demystify Transformer architecture, offering a clearer understanding of existing designs. This exploration aims to provide fresh insights and spark new avenues for Transformer innovation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。