证明Transformer可精确模拟任意注意力机制，无需训练数据。
On the Existence of Universal Simulators of Attention
- 构建通用模拟器$\u005cmathcal{U}$，用Transformer编码器实现注意力运算
- 通过RASP框架实现矩阵与激活操作的算法复现，无需依赖数据
- 首次给出不依赖学习的确定性解决方案，适用于理论研究者
以往关于Transformer可学习性的研究主要聚焦于通过训练逼近特定算法模式，但多为数据驱动，仅提供概率保证而非确定性解。相比之下，表达能力研究则从理论上探讨该架构可计算的问题，已证明Transformer的图灵完备性，并研究了电路复杂度与形式逻辑的界限。在可学习性与表达能力的交叉点上，核心问题仍存：\emph{Transformer作为计算模型，能否精确模拟任意注意力机制，尤其是其底层操作？}本研究探究Transformer编码器模拟基础注意力机制的能力。通过构建由变压器编码器组成的通用模拟器$\mathcal{U}$，我们提出算法方案，利用RASP（一种变压器计算的形式化框架），精确复现注意力输出及底层的矩阵与激活操作。结果表明，存在一种算法可达、数据无关的确定性解决方案，此前仅知可通过学习近似实现。
原文摘要 · Abstract (English)
Previous work on the learnability of transformers \textemdash\ focused primarily on examining their ability to approximate specific algorithmic patterns through training \textemdash\ has largely been data-driven, offering only probabilistic guarantees rather than deterministic solutions. Expressivity, on the contrary, has been devised to address the problems \emph{computable} by such architecture theoretically. These results proved the Turing-completeness of transformers, investigated bounds focused on circuit complexity, and formal logic. Being at the crossroad between learnability and expressivity, the question remains: \emph{can a transformer, as a computational model, simulate an arbitrary attention mechanism, or in particular, the underlying operations?} In this study, we investigate the transformer encoder's ability to simulate a vanilla attention mechanism. By constructing a universal simulator $\mathcal{U}$ composed of transformer encoders, we present algorithmic solutions to replicate attention outputs and the underlying elementary matrix and activation operations via RASP, a formal framework for transformer computation. We show the existence of an algorithmically achievable, data-agnostic solution, previously known to be approximated only by learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。