Transformer可近乎最优地完成线性分类元学习，仅需少量上下文示例。
Transformers are almost optimal metalearners for linear classification
- 用梯度下降训练简化Transformer，在共享低维子空间中学习任务模式。
- 新任务仅需 $O(k / R^4)$ 个示例即可泛化，接近理论最优性能。
- 适用于高维数据中的高效小样本元学习，对维度 $d$ 不敏感。
Transformers在上下文学习（ICL）中表现出色，引发其能否作为元学习器，在不额外训练的情况下仅用少量上下文示例适应新任务的疑问。现有理论分析多未涵盖正式的元学习设定——即通过联合解决相关任务提升效率。本文首次证明：在一类线性分类任务中，经梯度下降训练的简化Transformer可成为近似最优元学习器。每项任务对应一个类条件高斯混合模型，其均值向量位于 $R^d$ 中的 $k$-维共享子空间。训练足够多此类任务后，该Transformer在测试时仅需 $O(k / R^4)$ 个上下文示例即可泛化，几乎达到已知共享子空间的最优学习者性能，显著优于仅依赖上下文数据的学习者（需 $Ω(d / R^4)$ 示例）。关键的是，所需训练任务数和每任务示例数与环境维度 $d$ 无关。
原文摘要 · Abstract (English)
Transformers have demonstrated impressive in-context learning (ICL) capabilities, raising the question of whether they can serve as metalearners that adapt to new tasks using only a small number of in-context examples, without any further training. While recent theoretical work has studied transformers' ability to perform ICL, most of these analyses do not address the formal metalearning setting, where the objective is to solve a collection of related tasks more efficiently than would be possible by solving each task individually. In this paper, we provide the first theoretical analysis showing that a simplified transformer architecture trained via gradient descent can act as a near-optimal metalearner in a linear classification setting. We consider a natural family of tasks where each task corresponds to a class-conditional Gaussian mixture model, with the mean vectors lying in a shared $k$-dimensional subspace of $R^d$. After training on a sufficient number of such tasks, we show that the transformer can generalize to a new task using only $O(k / R^4)$ in-context examples, where $R$ denotes the signal strength at test time. This performance (almost) matches that of an optimal learner that knows exactly the shared subspace and significantly outperforms any learner that only has access to the in-context data, which requires $Ω(d / R^4)$ examples to generalize. Importantly, our bounds on the number of training tasks and examples per task needed to achieve this result are independent of the ambient dimension $d$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。