将Transformer解释为概率图谱嵌入的迭代推断，可提升模型性能。
Transformers as Unrolled Inference in Probabilistic Laplacian Eigenmaps: An Interpretation and Potential Improvements
- 从概率图谱嵌入视角重解Transformer，发现其初始阶段等价于线性降维。
- 在变换器块中推导出图拉普拉斯项，而非传统注意力矩阵。
- 减去注意力矩阵中的单位矩阵可提升语言与视觉模型验证效果。
我们提出一种将Transformer视为概率拉普拉斯特征映射模型中未展开的推断步骤的概率解释，该模型源自ProbDR框架。推导表明，在初始化时，Transformer执行的是‘线性’降维。此外，我们证明在Transformer模块内部，图拉普拉斯项由我们的论证自然产生，而非注意力矩阵（我们将其解释为邻接矩阵）。实验显示，仅对注意力矩阵减去单位矩阵（即执行一次图扩散步骤），即可在语言模型和简单视觉Transformer上提升验证性能。
原文摘要 · Abstract (English)
We propose a probabilistic interpretation of transformers as unrolled inference steps assuming a probabilistic Laplacian Eigenmaps model from the ProbDR framework. Our derivation shows that at initialisation, transformers perform "linear" dimensionality reduction. We also show that within the transformer block, a graph Laplacian term arises from our arguments, rather than an attention matrix (which we interpret as an adjacency matrix). We demonstrate that simply subtracting the identity from the attention matrix (and thereby taking a graph diffusion step) improves validation performance on a language model and a simple vision transformer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。