语言模型解码本质可实现通用计算，训练只提升编程易用性。
Universal computation is intrinsic to language model decoding
- 通过链式自回归输出实现通用计算，等价于图灵机
- 随机初始化模型已具备通用计算能力，无需训练
- 揭示训练本质是提升自然语言编程效率而非计算能力
语言模型如今可通过自然语言表达并解决各类通用问题，但其终极计算能力仍是科学界争论焦点。与传统计算机不同，语言模型通过自回归方式预测人类生成文本的后续元素。本文证明，将语言模型的自回归输出串联起来足以实现通用计算——即模拟任意算法在任意输入上的执行过程。因此，激发期望计算行为的挑战可重新理解为可编程性问题：即找到合适提示的难易程度。令人震惊的是，我们展示了即使在随机初始化状态下，语言模型也具备通用计算能力。这意味着训练并未赋予模型计算表现力，而是提升了其可编程性，使得自然语言成为访问这些内在能力的有效接口。
原文摘要 · Abstract (English)
Language models now provide an interface to express and often solve general problems in natural language, yet their ultimate computational capabilities remain a major topic of scientific debate. Unlike a formal computer, a language model is trained to autoregressively predict successive elements in human-generated text. We prove that chaining a language model's autoregressive output is sufficient to perform universal computation. That is, a language model can simulate the execution of any algorithm on any input. The challenge of eliciting desired computational behaviour can thus be reframed in terms of programmability: the ease of finding a suitable prompt. Strikingly, we demonstrate that even randomly initialized language models are capable of universal computation before training. This implies that training does not give rise to computational expressiveness -- rather, it improves programmability, enabling a natural language interface for accessing these intrinsic capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。