证明了仅用注意力机制就能逼近任意序列映射，无需传统网络结构。
Universal Approximation with Softmax Attention
- 通过新插值方法揭示注意力内部机制，实现通用逼近。
- 两层自注意力或单层加Softmax即可任意逼近连续序列函数。
- 适用于理解Transformer理论本质的研究者，尤其关注注意力机制者。
我们证明，在紧致域上，仅使用线性变换的两层自注意力，以及一层自注意力后接Softmax函数，均可作为连续序列到序列函数的通用逼近器。核心方法是一种基于插值的新技术，用于分析注意力的内部机制。关键发现是：自注意力能以任意精度逼近广义ReLU函数，从而涵盖多种已知的通用逼近器。基于此，我们进一步证明，仅需两层多头注意力即可构成序列到序列的通用逼近器。相比之下，以往工作依赖前馈网络来建立Transformer的通用逼近性。此外，我们的方法还可扩展至证明（Softmax-）注意力层能够逼近各类上下文中的统计模型。这些技术本身具有独立研究价值。
原文摘要 · Abstract (English)
We prove that with linear transformations, both (i) two-layer self-attention and (ii) one-layer self-attention followed by a softmax function are universal approximators for continuous sequence-to-sequence functions on compact domains. Our main technique is a new interpolation-based method for analyzing attention's internal mechanism. This leads to our key insight: self-attention is able to approximate a generalized version of ReLU to arbitrary precision, and hence subsumes many known universal approximators. Building on these, we show that two-layer multi-head attention alone suffices as a sequence-to-sequence universal approximator. In contrast, prior works rely on feed-forward networks to establish universal approximation in Transformers. Furthermore, we extend our techniques to show that, (softmax-)attention-only layers are capable of approximating various statistical models in-context. We believe these techniques hold independent interest.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。