揭示Transformer如何通过注意力机制实现非线性回归的上下文学习
Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizer
- 用注意力机制构建多项式/样条等非线性特征，实现端到端非线性回归
- 给出有限样本下的泛化误差界，依赖上下文长度和训练集大小
- 理论可解释性强，适合研究ICL机制的学者与模型设计者
预训练Transformer能够在不更新权重的情况下，仅通过提示中的示例进行学习，这种能力被称为上下文学习（ICL）。尽管ICL在多个领域表现优异，但其理论理解仍不充分。现有理论多集中于线性模型，本文研究非线性回归场景下的ICL。通过注意力机制的交互，我们显式构造了能实现多项式或样条基等非线性特征的Transformer网络，覆盖广泛函数类。基于此构造，建立了分析端到端上下文非线性回归的理论框架，给出了关于上下文长度和训练集大小的有限样本泛化误差界。我们在合成回归任务上数值验证了该理论的有效性。
原文摘要 · Abstract (English)
Pre-trained transformers are able to learn from examples provided as part of the prompt without any weight updates, a remarkable ability known as in-context learning (ICL). Despite its demonstrated efficacy across various domains, the theoretical understanding of ICL is still developing. Whereas most existing theory has focused on linear models, we study ICL in the nonlinear regression setting. Through the interaction mechanism in attention, we explicitly construct transformer networks to realize nonlinear features, such as polynomial or spline bases, which span a wide class of functions. Based on this construction, we establish a framework to analyze end-to-end in-context nonlinear regression with the constructed features. Our theory provides finite-sample generalization error bounds in terms of context length and training set size. We numerically validate the theory on synthetic regression tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。