实证表明,Transformer的计算能力严重依赖上下文管理机制。
Position: The Turing-Completeness of Autoregressive Transformers Relies Heavily on Context Management
- 区分固定模型与扩展模型两种设置,明确真实大模型运行场景
- 不同上下文管理方法使模型计算能力差异显著
- 强调上下文管理是决定Transformer实际计算能力的核心
许多研究声称Transformer具有图灵完备性,但文献常混淆两种不同设定:(i) 固定系统设置,即固定自回归Transformer配合固定上下文管理方法逐步处理不同长度输入;(ii) 扩展家族设置,即使用一系列不同规模模型(增大上下文窗口或数值精度)来处理不同长度输入。现有图灵完备性证明多基于设定(ii),而真实大模型部署及标准图灵完备性概念更符合设定(i)。本文首先形式化固定系统设置,准确刻画真实大模型运行机制;接着指出扩展家族设定下的结果虽提供理论资源界,却无法证明图灵完备性,澄清了对已有结论的常见误解;最后证明不同上下文管理方法导致计算能力差异显著,主张上下文管理是决定真实自回归Transformer计算能力的核心组件。
原文摘要 · Abstract (English)
Many works make the eye-catching claim that Transformers are Turing-complete. However, the literature often conflates two distinct settings: (i) a fixed Transformer system setting, in which a fixed autoregressive Transformer is coupled with a fixed context-management method to process inputs of different lengths step by step, and (ii) a scaling-family setting, in which a family of different models (with increasing context-window length or numerical precision) is used to handle different input lengths. Existing proofs of Transformer Turing-completeness are frequently established in setting (ii), whereas real-world LLM deployment and the standard notion of Turing-completeness correspond more naturally to setting (i). In this paper, we first formalize the fixed-system setting, thereby providing a concrete characterization of how real-world LLMs operate. We then argue that results proved in the scaling-family setting provide theoretically meaningful resource bounds but do not establish Turing-completeness, thereby clarifying a common misinterpretation of existing results. Finally, we show that different context-management methods can yield sharply different computational power, and we advocate the position that context management is a central component that critically determines the computational power of real-world autoregressive Transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。