语言模型通过自回归生成可实现通用计算，无需修改权重。
Autoregressive Large Language Models are Computationally Universal
- 用扩展自回归解码模拟拉格系统，突破上下文长度限制。
- 证明2027条规则的图灵机可被该系统模拟。
- 实测Gemini-1.5-Pro在提示下可正确执行全部规则。
我们证明基于Transformer的语言模型在自回归解码下可实现通用计算，无需外部干预或修改模型权重。为处理任意长输入，提出一种新解码机制：随上下文窗口推进，生成的标记被追加至序列末尾。该机制对应经典计算模型——拉格系统，已知其具有通用计算能力。通过新证明，我们展示一个包含2027条产生规则的通用图灵机可被拉格系统模拟。进一步验证现有大模型：仅用单一系统提示，gemini-1.5-pro-001 在确定性（贪婪）解码下能正确执行全部2027条规则。根据邱奇-图灵论题，此配置下的Gemini-1.5-Pro具备通用计算机功能。
原文摘要 · Abstract (English)
We show that autoregressive decoding of a transformer-based language model can realize universal computation, without external intervention or modification of the model's weights. Establishing this result requires understanding how a language model can process arbitrarily long inputs using a bounded context. For this purpose, we consider a generalization of autoregressive decoding where, given a long input, emitted tokens are appended to the end of the sequence as the context window advances. We first show that the resulting system corresponds to a classical model of computation, a Lag system, that has long been known to be computationally universal. By leveraging a new proof, we show that a universal Turing machine can be simulated by a Lag system with 2027 production rules. We then investigate whether an existing large language model can simulate the behaviour of such a universal Lag system. We give an affirmative answer by showing that a single system-prompt can be developed for gemini-1.5-pro-001 that drives the model, under deterministic (greedy) decoding, to correctly apply each of the 2027 production rules. We conclude that, by the Church-Turing thesis, prompted gemini-1.5-pro-001 with extended autoregressive (greedy) decoding is a general purpose computer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。