首次给出Transformer对霍尔德函数逼近误差的上下界，揭示其理论极限。
Approximation Error Upper and Lower Bounds for Hölder Class with Transformers
- 构建标准Transformer在霍尔德函数上的逼近上界，证明所需层数为O(ε^{-d₀/α})
- 基于VC维推导下界，证明至少需Ω(ε^{-d₀/(4α)})层才能达到ε精度
- 将理论结果拓展至回归任务，验证其在实际场景中的有效性
我们通过建立霍尔德类函数的精确逼近误差上下界，探讨了Transformer的表达能力。具体而言，针对配备Softmax、ReLU和残差连接的标准Transformer架构，推导出新的逼近上界：当输入维度为d₀、光滑度为α∈(0,1]时，最多使用O(ε^{-d₀/α})个模块即可在任意精度ε>0下逼近任意有界霍尔德函数。在下界方面，利用VC维上界，首次严格证明：要实现ε逼近精度，Transformer至少需要Ω(ε^{-d₀/(4α)})个模块。最后，我们将上述结果推广至一般回归任务，建立了相应的超出风险率，展示了Transformer在真实场景中的实证有效性。
原文摘要 · Abstract (English)
We explore the expressive power of Transformers by establishing precise approximation error upper and lower bounds for Hölder class. Specifically, a new approximation upper bound is derived for the standard Transformer architecture equipped with Softmax operators, ReLU activation functions, and residual connections. We prove that a Transformer network composed of at most $\mathcal{O}(\varepsilon^{-{d_{0}}/α})$ blocks can approximate any bounded Hölder function with $d_{0}$-dimensional input and smoothness $α\in(0,1]$ under any accuracy $\varepsilon>0$. In the case of approximation lower bounds, leveraging the VC-dimension upper bound, we are the first to rigorously prove that Transformers demand for at least $Ω(\varepsilon^{-{d_{0}}/({4α})})$ blocks to achieve the $\varepsilon$ approximation accuracy. As a final step, we extend the derived results for standard Transformers to a general regression task and establish the corresponding excess risk rates demonstrating Transformers' empirical effectiveness in real-world settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。