浮点实现让Transformer能表达非置换不变函数，但长序列下能力受限。
On the Expressive Power of Floating-Point Transformers
- 在浮点数运算下，Transformer可表示非置换不变函数，无需位置编码。
- 序列长度较短时可逼近所有置换不变函数，长序列则无法做到。
- 非平凡的加性位置编码反而降低模型表达能力，需谨慎设计。
现有研究表明，基于实数参数和精确运算的Transformer具有置换等变性，且可逼近任意置换等变连续函数。然而实际计算中使用的是有限浮点数和存在舍入误差的机器运算。本文研究浮点数参数与浮点运算下的Transformer表达能力。不同于精确运算下的结论，我们首次证明：浮点变压器即使无位置编码，也能表示一类非置换等变函数。当序列长度有界时，其可表示所有置换等变函数；但序列过长时则不可。此外，我们揭示了浮点变压器中的最小等变结构，并发现所有非平凡的加性位置编码都会损害其表达能力。
原文摘要 · Abstract (English)
The study on the expressive power of transformers shows that transformers are permutation equivariant, and they can approximate all permutation-equivariant continuous functions on a compact domain. However, these results are derived under real parameters and exact operations, while real implementations on computers can only use a finite set of numbers and inexact machine operations with round-off errors. In this work, we investigate the representability of floating-point transformers that use floating-point parameters and floating-point operations. Unlike existing results under exact operations, we first show that floating-point transformers can represent a class of non-permutation-equivariant functions even without positional encoding. Furthermore, we prove that floating-point transformers can represent all permutation-equivariant functions when the sequence length is bounded, but they cannot when the sequence length is large. We also found the minimal equivariance structure in floating-point transformers, and show that all non-trivial additive positional encoding can harm the representability of floating-point transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。