固定权重的Transformer可通过提示实现多种算法模拟,无需修改参数。
In-Context Algorithm Emulation in Fixed-Weight Transformers
- 用提示编码算法参数,靠注意力机制实现计算路径
- 单层注意力可精确模拟任意实函数,涵盖梯度下降等经典算法
- 无需更新模型,仅靠提示就能切换不同算法,适合快速部署
我们证明,一个权重固定的最小化Transformer可通过上下文提示模拟一类广泛存在的算法。我们形式化了两种上下文算法模拟模式:在任务特定模式下,对于任意连续函数 $f: \mathbb{R} \to \mathbb{R}$,存在一个单头softmax注意力层,其前向传播可对形如 $f(w^\top x - y)$ 的函数以任意精度重现。该通用模板涵盖了众多主流机器学习算法(如梯度下降、线性回归、岭回归)。在提示可编程模式下,我们证明了普遍性:一个固定权重的两层softmax注意力模块,仅通过提示即可模拟所有任务特定类中的算法(即每个算法均可由单个softmax注意力实现)。核心思想是构造能将算法参数编码到标记表示中的提示,产生明显的点积间隙,迫使softmax注意力遵循预期计算路径。该方法无需前馈层,也无需参数更新,所有适应均通过提示完成。数值结果验证了理论。这些发现建立了上下文学习与算法模拟之间的直接联系,为大模型提供了一种仅通过提示即可调用算法库的简单机制,揭示了GPT类基础模型仅凭提示即可切换算法的可能性,并在现代Transformer模型中确立了算法普遍性。
原文摘要 · Abstract (English)
We prove that a minimal Transformer with frozen weights emulates a broad class of algorithms by in-context prompting. We formalize two modes of in-context algorithm emulation. In the task-specific mode, for any continuous function $f: \mathbb{R} \to \mathbb{R}$, we show the existence of a single-head softmax attention layer whose forward pass reproduces functions of the form $f(w^\top x - y)$ to arbitrary precision. This general template subsumes many popular machine learning algorithms (e.g., gradient descent, linear regression, ridge regression). In the prompt-programmable mode, we prove universality: a single fixed-weight two-layer softmax attention module emulates all algorithms from the task-specific class (i.e., each implementable by a single softmax attention) via only prompting. Our key idea is to construct prompts that encode an algorithm's parameters into token representations, creating sharp dot-product gaps that force the softmax attention to follow the intended computation. This construction requires no feed-forward layers and no parameter updates. All adaptation happens through the prompt alone. Numerical results corroborate our theory. These findings forge a direct link between in-context learning and algorithmic emulation, and offer a simple mechanism for large Transformers to serve as prompt-programmable libraries of algorithms. They illuminate how GPT-style foundation models may swap algorithms via prompts alone, and establish a form of algorithmic universality in modern Transformer models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。