揭示Transformer如何通过注意力机制实现分段线性函数的高效表达
On the Expressive Power of Transformers for Maxout Networks and Continuous Piecewise Linear Functions
- 用Transformer逼近maxout网络，保持模型复杂度不变
- 深度增加使线性区域数量呈指数增长，表达能力大幅提升
- 解析注意力与前馈层的分工：注意力做最大值运算，前馈做仿射变换
Transformer在众多应用中表现出色，但其理论表达能力仍不清晰。本文首次建立Transformer与maxout网络之间的显式近似关系，在保持相似模型复杂度的前提下，证明了Transformer继承了ReLU网络的通用逼近能力。基于此，我们构建了分析连续分段线性函数逼近的框架，定量刻画其表达力：线性区域数随深度呈指数增长。该研究建立了标准前馈神经网络逼近理论与Transformer架构间的理论桥梁，并揭示结构本质：自注意力层实现max型操作，前馈层完成逐标记仿射变换。
原文摘要 · Abstract (English)
Transformer networks have achieved remarkable empirical success across a wide range of applications, yet their theoretical expressive power remains insufficiently understood. In this paper, we study the expressive capabilities of Transformer architectures. We first establish an explicit approximation of maxout networks by Transformer networks while preserving comparable model complexity. As a consequence, Transformers inherit the universal approximation capability of ReLU networks under similar complexity constraints. Building on this connection, we develop a framework to analyze the approximation of continuous piecewise linear functions by Transformers and quantitatively characterize their expressivity via the number of linear regions, which grows exponentially with depth. Our analysis establishes a theoretical bridge between approximation theory for standard feedforward neural networks and Transformer architectures. It also yields structural insights into Transformers: self-attention layers implement max-type operations, while feedforward layers realize token-wise affine transformations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。