小规模Transformer学会执行通用编程语言，可泛化到人类编写的复杂程序。
Training Transformers as a Universal Computer
- 用PENCIL框架实现空间高效的微步执行，适应有限上下文窗口
- 训练后能正确执行位复制、二进制加乘及SAT求解等任务
- 在分布外程序上展现泛化能力，验证Transformer可作通用计算机
我们证明，一个小规模Transformer可以学习执行MicroPy——一种简化但计算完备的编程语言。给定过程定义和待求值表达式，Transformer利用PENCIL框架进行小步执行，实现在有限上下文窗口内的高效计算。在随机生成的无意义MicroPy程序上训练后，该模型能泛化至多种人工编写的程序，包括位复制与翻转、二进制加法与乘法、以及SAT验证与求解。值得注意的是，模型具备分布外泛化能力，即能处理未见过的同分布程序。由于MicroPy可表达任意计算，本结果为标准Transformer可被训练为通用计算机提供了实证支持。
原文摘要 · Abstract (English)
We demonstrate that a small transformer can learn to execute programs in MicroPy, a simplified yet computationally universal programming language. Given procedure definitions together with an expression to evaluate, the transformer predicts small-step execution using PENCIL scaffolding for space-efficient execution within a bounded context window. After training on randomly generated, meaningless MicroPy programs, the learned transformer generalizes to various human-written programs including bit copying and flipping, binary addition and multiplication, and SAT verification and solving. We note that the trained model can achieve out-of-distribution generalization; i.e., evaluate novel programs from distribution on programs. Since MicroPy can express any computation, our results provide empirical evidence that a standard transformer can be trained to act as a universal computer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。