Transformer本质是无状态的可微神经计算机。
Transformers are Stateless Differentiable Neural Computers
- 将Transformer视为无内部状态的可微神经计算机，内存为一次性写入矩阵。
- 多头注意力对应多个并行读头，键值寻址即内容寻址机制。
- 统一解释自注意力与交叉注意力，适合研究模型计算本质者阅读。
可微神经计算机（DNC）是一类带可寻址外部内存的循环架构，支持可微读写操作。相比之下，Transformer通常被视为基于多头自注意力的前馈架构。本文通过形式推导表明，因果Transformer层恰好等同于无状态可微神经计算机（sDNC）：(1) 控制器无递归内部状态；(2) 外部内存为一次性写入的价值向量矩阵；(3) 通过键实现内容寻址；(4) 多头注意力对应多个并行读头。我们进一步将该等价性扩展至交叉注意力，证明编码器-解码器Transformer正是具有独立读取与写入内存的sDNC。本工作为Transformer提供了统一的内存中心解释，有助于将现代大语言模型置于严谨的计算框架中。
原文摘要 · Abstract (English)
Differentiable Neural Computers (DNCs) were introduced as recurrent architectures equipped with an addressable external memory supporting differentiable read and write operations. Transformers, in contrast, are nominally feedforward architectures based on multi-head self-attention. In this work we give a formal derivation showing that a causal Transformer layer is exactly a stateless Differentiable Neural Computer (sDNC) where (1) the controller has no recurrent internal state, (2) the external memory is a write-once matrix of value vectors, (3) content-based addressing via keys implements attention, and (4) multi-head attention corresponds to multiple parallel read heads. We further extend this equivalence to cross-attention, showing that encoder-decoder Transformers are precisely sDNCs with distinct read-from and write-to memories. Our results provide a unified memory-centric interpretation of Transformers and contribute to the ongoing effort to place modern large language models in a principled computational framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。