单层Transformer能学会最近邻分类，无需微调。
One-Layer Transformer Provably Learns One-Nearest Neighbor In Context
- 用单层softmax注意力实现最近邻预测规则
- 梯度下降可训练非凸损失函数并收敛到正确行为
- 为Transformer的上下文学习机制提供理论解释
近年来，Transformer取得了巨大成功。有趣的是，Transformer展现出强大的上下文学习能力——即使不进行微调，仅通过任务提示也能有效解决未见任务。本文研究单层Transformer学习最经典的非参数估计器之一：一近邻预测规则的能力。在提示包含一系列带标签训练数据和无标签测试数据的理论框架下，我们证明：尽管损失函数在梯度下降训练中是非凸的，但单个softmax注意力层仍能成功学习表现出一近邻分类器的行为。该结果为Transformer如何被训练以实现非参数机器学习算法提供了具体实例，并揭示了softmax注意力在Transformer模型中的作用。
原文摘要 · Abstract (English)
Transformers have achieved great success in recent years. Interestingly, transformers have shown particularly strong in-context learning capability -- even without fine-tuning, they are still able to solve unseen tasks well purely based on task-specific prompts. In this paper, we study the capability of one-layer transformers in learning one of the most classical nonparametric estimators, the one-nearest neighbor prediction rule. Under a theoretical framework where the prompt contains a sequence of labeled training data and unlabeled test data, we show that, although the loss function is nonconvex when trained with gradient descent, a single softmax attention layer can successfully learn to behave like a one-nearest neighbor classifier. Our result gives a concrete example of how transformers can be trained to implement nonparametric machine learning algorithms, and sheds light on the role of softmax attention in transformer models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。