揭示变换器对列对称多项式的逼近能力，明确网络规模与表达力的关系。
Approximation of Permutation Invariant Polynomials by Transformers: Efficient Construction in Column-Size
- 基于对称多项式代数性质，设计高效构造方法
- 证明变换器大小与逼近精度存在显式关系
- 为模型参数效率提供理论支持，适合研究架构设计者
Transformer 是一类在多个领域表现出色的神经网络，尤其在自然语言处理中表现突出。受此启发，学术界对 Transformer 的理论理解日益关注。其中一项重要进展是对其近似能力的数学分析，验证了其经验上的表达能力。本文研究 Transformer 逼近列对称多项式的能力——这是以矩阵为输入的对称多项式的推广。通过利用变换器的参数效率及其与对称性的兼容性，结合对称多项式的代数特性，我们建立了变换器网络规模与其近似能力之间的显式关系。
原文摘要 · Abstract (English)
Transformers are a type of neural network that have demonstrated remarkable performance across various domains, particularly in natural language processing tasks. Motivated by this success, research on the theoretical understanding of transformers has garnered significant attention. A notable example is the mathematical analysis of their approximation power, which validates the empirical expressive capability of transformers. In this study, we investigate the ability of transformers to approximate column-symmetric polynomials, an extension of symmetric polynomials that take matrices as input. Consequently, we establish an explicit relationship between the size of the transformer network and its approximation capability, leveraging the parameter efficiency of transformers and their compatibility with symmetry by focusing on the algebraic properties of symmetric polynomials.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。