证明单头注意力可逼近任意连续函数，为模型设计提供理论依据
Attention Mechanism, Max-Affine Partition, and Universal Approximation
- 将注意力视为输入区域划分机制，通过设计权重模拟目标函数
- 单层自注意力加线性变换可逼近任意连续函数（$L_\infty$范数下）
- 首次实现单头交叉注意力的通用逼近能力，适用于复杂函数建模
我们建立了单层、单头自注意力和交叉注意力机制在极简附加结构下的通用逼近能力。核心洞察是将单头注意力解释为对输入域进行分块并为各子区域分配不同值的机制。基于此，我们设计注意力权重，使其能够模仿目标函数。由此证明，在 $L_\infty$-范数下，单层自注意力前接线性求和变换即可逼近任意紧集上的连续函数。进一步地，该构造可扩展至 $L_p$-范数（$1\leq p <\infty$），实现对任意勒贝格可积函数的逼近。最后，我们还将该方法拓展至单头交叉注意力，首次证明其具备相同的通用逼近能力。
原文摘要 · Abstract (English)
We establish the universal approximation capability of single-layer, single-head self- and cross-attention mechanisms with minimal attached structures. Our key insight is to interpret single-head attention as an input domain-partition mechanism that assigns distinct values to subregions. This allows us to engineer the attention weights such that this assignment imitates the target function. Building on this, we prove that a single self-attention layer, preceded by sum-of-linear transformations, is capable of approximating any continuous function on a compact domain under the $L_\infty$-norm. Furthermore, we extend this construction to approximate any Lebesgue integrable function under $L_p$-norm for $1\leq p <\infty$. Lastly, we also extend our techniques and show that, for the first time, single-head cross-attention achieves the same universal approximation guarantees.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。