将自注意力机制解释为无反向传播的吸引子网络，实现瞬时记忆。
Self-attention as an attractor network: transient memories without backpropagation
- 用局部能量项替代注意力计算，类比伪似然构建动态系统。
- 模型在训练与测试样本间产生强关联的瞬时状态，无需梯度更新。
- 为理解Transformer提供物理启发的新框架，适合理论研究者。
Transformers是现代神经网络中最成功的架构之一。其核心是注意力机制，近年来引起物理界关注，因其在某些情况下可表示为能量函数的导数：尽管交叉注意力层可视为现代霍普菲尔德网络，但自注意力层（用于GPT等自回归模型）却无法如此表达。本文证明，自注意力层可作为局部能量项的导数，其形式类似伪似然。我们利用该类比设计了一种无需反向传播的循环模型，其动态过程展现出与训练和测试样本高度相关的瞬时状态。整体上，我们提出了一个新框架，将自注意力解释为吸引子网络，或为受物理学启发理解Transformer开辟新路径。
原文摘要 · Abstract (English)
Transformers are one of the most successful architectures of modern neural networks. At their core there is the so-called attention mechanism, which recently interested the physics community as it can be written as the derivative of an energy function in certain cases: while it is possible to write the cross-attention layer as a modern Hopfield network, the same is not possible for the self-attention, which is used in the GPT architectures and other autoregressive models. In this work we show that it is possible to obtain the self-attention layer as the derivative of local energy terms, which resemble a pseudo-likelihood. We leverage the analogy with pseudo-likelihood to design a recurrent model that can be trained without backpropagation: the dynamics shows transient states that are strongly correlated with both train and test examples. Overall we present a novel framework to interpret self-attention as an attractor network, potentially paving the way for new theoretical approaches inspired from physics to understand transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。