研究大模型如何在上下文中学习线性函数，发现其泛化能力有限。
Re-examining learning linear functions in context
- 用合成数据测试Transformer模型的上下文学习能力
- 模型无法超越训练分布，泛化性能差
- 挑战了模型会学线性回归的主流看法，适合关注模型机制的研究者
上下文学习（ICL）已成为大型语言模型（LLMs）快速适配各类任务的强大范式。然而，我们对ICL的工作原理仍理解不足。本文在可控环境下，使用合成训练数据，研究单变量线性函数的上下文学习。实验采用从零训练的一系列类似GPT-2的Transformer模型。结果挑战了当前普遍认为Transformer会采用类似线性回归的算法来实现上下文学习的观点。这些模型在训练分布之外无法泛化，暴露出其推断抽象任务结构能力的根本局限。我们的实验促使我们提出一个数学上精确的假设，关于模型实际学习的内容。
原文摘要 · Abstract (English)
In-context learning (ICL) has emerged as a powerful paradigm for easily adapting Large Language Models (LLMs) to various tasks. However, our understanding of how ICL works remains limited. We explore a simple model of ICL in a controlled setup with synthetic training data to investigate ICL of univariate linear functions. We experiment with a range of GPT-2-like transformer models trained from scratch. Our findings challenge the prevailing narrative that transformers adopt algorithmic approaches like linear regression to learn a linear function in-context. These models fail to generalize beyond their training distribution, highlighting fundamental limitations in their capacity to infer abstract task structures. Our experiments lead us to propose a mathematically precise hypothesis of what the model might be learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。