提示词让小模型也能做任何计算，理论证明其通用性。
Ask, and it shall be given: On the Turing completeness of prompting
- 用有限大小的Transformer和提示词实现任意可计算函数
- 单个模型在复杂度上接近无限大小模型的表现
- 为提示工程提供理论基础，适合研究者和工程师参考
自GPT成功以来，大语言模型（LLMs）正重塑机器学习，并催生了提示工程范式。人们训练一个通用的大型语言模型，通过不同提示完成各类任务。然而，这种实践的成功缺乏理论支撑。本文首次对提示范式进行理论分析，证明提示具有图灵完备性：存在一个有限大小的Transformer，使得对任意可计算函数，均存在一个对应提示，使该Transformer能计算该函数。此外，即使仅使用单一有限大小的Transformer，其复杂度边界仍几乎等同于所有无界大小Transformer的类。结果表明，提示可使单一有限大小Transformer高效实现通用计算，为实际中的提示工程提供了理论依据。
原文摘要 · Abstract (English)
Since the success of GPT, large language models (LLMs) have been revolutionizing machine learning and have initiated the so-called LLM prompting paradigm. In the era of LLMs, people train a single general-purpose LLM and provide the LLM with different prompts to perform different tasks. However, such empirical success largely lacks theoretical understanding. Here, we present the first theoretical study on the LLM prompting paradigm to the best of our knowledge. In this work, we show that prompting is in fact Turing-complete: there exists a finite-size Transformer such that for any computable function, there exists a corresponding prompt following which the Transformer computes the function. Furthermore, we show that even though we use only a single finite-size Transformer, it can still achieve nearly the same complexity bounds as that of the class of all unbounded-size Transformers. Overall, our result reveals that prompting can enable a single finite-size Transformer to be efficiently universal, which establishes a theoretical underpinning for prompt engineering in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。