Transformer自注意力在高置信度下等价于最短路径算法，揭示思维链的几何本质。
The Geometry of Thought: Disclosing the Transformer as a Tropical Polynomial Circuit
- 将softmax注意力取热带极限，转化为最大-加法矩阵乘积
- 前向传播等价于在隐空间图上执行动态规划路径搜索
- 为思维链推理提供最短路径计算的理论解释，适合理论研究者
我们证明，在高置信度情形（β→∞，其中β为逆温度）下，Transformer自注意力机制在热带半环（max-plus代数）中运行。具体而言，将softmax注意力取热带极限后，其转化为热带矩阵乘积。这表明Transformer前向传播实质上是在由标记相似性定义的隐空间图上执行动态规划递推（即贝尔曼-福特路径查找更新）。该理论结果为思维链推理提供了新的几何视角：其源于网络内部固有的最短路径（或最长路径）算法的执行。
原文摘要 · Abstract (English)
We prove that the Transformer self-attention mechanism in the high-confidence regime ($β\to \infty$, where $β$ is an inverse temperature) operates in the tropical semiring (max-plus algebra). In particular, we show that taking the tropical limit of the softmax attention converts it into a tropical matrix product. This reveals that the Transformer's forward pass is effectively executing a dynamic programming recurrence (specifically, a Bellman-Ford path-finding update) on a latent graph defined by token similarities. Our theoretical result provides a new geometric perspective for chain-of-thought reasoning: it emerges from an inherent shortest-path (or longest-path) algorithm being carried out within the network's computation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。