一个固定模型能模拟任意同类型Transformer，靠输入编码实现。
Fixed Universal Transformers
- 用输入嵌入编码目标模型，内部参数不变即可实现通用性
- 当嵌入维数足够大时，可构造出具备通用性的稀疏结构
- 随机初始化的Transformer几乎必然具备通用性，适合理论研究
我们提出𝑁𝑡𝑲𝑰𝑴𝒂𝒔𝒋𝒎（universal transformers）：通过合适的输入嵌入，固定参数的Transformer可模拟给定类别中的任意Transformer。类似通用图灵机，输入嵌入编码目标模型描述，内部参数保持不变。我们给出显式的稀疏构造，在嵌入维数足够大时实现通用性；进一步证明通用性是普遍现象：随机初始化的Transformer几乎必然具有通用性，与Zhong和Andreas（2024）的近期实证结果一致。我们在括号匹配和多跳推理等算法任务上验证了该理论。结果表明，Transformer的表达能力可能更多来自输入表示而非学习权重。
原文摘要 · Abstract (English)
We introduce \emph{universal transformers}: fixed transformers that can simulate any transformer in a given class via a suitable input embedding. Analogous to a universal Turing machine, the input embedding encodes a description of the target model while all internal parameters remain fixed. We provide explicit sparse constructions achieving universality when the embedding dimension is sufficiently large, and further show that universality is generic: randomly initialized transformers are universal almost surely, which aligns with recent empirical results of Zhong and Andreas (2024). We empirically validate our theory on the algorithmic tasks of parenthesis balancing and multi-hop reasoning. Our results suggest that much of a transformer's expressive power may reside in its input representation rather than its learned weights.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。