Transformer可看作在全连接图上运行的GNN，用注意力机制学习元素间关系。
Transformers are Graph Neural Networks
- 将Transformer视为在全连接节点图上的消息传递GNN，自注意力决定节点重要性
- 位置编码提供序列顺序提示，使模型能捕捉结构信息
- 虽理论等价于GNN，但因密集运算更适配现代硬件，性能更优
本文建立了Transformer架构（最初用于自然语言处理）与图神经网络（GNN）在图表示学习中的联系。我们证明，Transformer可被视作在完全连接的词元图上运行的消息传递GNN，其中自注意力机制捕获所有词元之间的相对重要性，而位置编码则提供序列顺序或结构的提示。因此，Transformer是一种表达能力强的集合处理网络，能在不依赖预设图结构的情况下学习输入元素间的复杂关系。尽管在数学上与GNN等价，但由于采用密集矩阵运算，在现代硬件上比稀疏消息传递更高效。这揭示了：当前的Transformer实际上是赢在硬件红利上的GNN。
原文摘要 · Abstract (English)
We establish connections between the Transformer architecture, originally introduced for natural language processing, and Graph Neural Networks (GNNs) for representation learning on graphs. We show how Transformers can be viewed as message passing GNNs operating on fully connected graphs of tokens, where the self-attention mechanism capture the relative importance of all tokens w.r.t. each-other, and positional encodings provide hints about sequential ordering or structure. Thus, Transformers are expressive set processing networks that learn relationships among input elements without being constrained by apriori graphs. Despite this mathematical connection to GNNs, Transformers are implemented via dense matrix operations that are significantly more efficient on modern hardware than sparse message passing. This leads to the perspective that Transformers are GNNs currently winning the hardware lottery.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。