Transformer通过梯度下降实现上下文逻辑回归，揭示其学习机制。
Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descent
- 每层执行一步归一化梯度下降，多层协同完成逻辑回归。
- 训练收敛且在分布外数据上表现稳定，验证有效性。
- 适合研究大模型内学习机制的学者参考。
Transformer 在上下文学习（ICL）中表现出色，通常认为其能隐式执行算法以提升预测能力。本文研究了使用 softmax 注意力的 Transformer 在线性分类任务上的上下文学习表现。我们构建了一类多层 Transformer，能够执行上下文逻辑回归，其中每一层恰好执行一次对上下文损失的归一化梯度下降。进一步证明，该模型可通过（i）用单步梯度下降监督训练一个自注意力层，以及（ii）循环应用该层形成闭环模型来获得。本文提供了自注意力层的训练收敛保证，以及闭环模型的分布外泛化保证。结果深化了对 Transformer 上下文学习机制的理论理解，展示了 softmax Transformer 如何有效充当上下文学习者。
原文摘要 · Abstract (English)
Transformers have demonstrated remarkable in-context learning (ICL) capabilities. The strong ICL performance of transformers is commonly believed to arise from their ability to implicitly execute certain algorithms on the context, thereby enhancing prediction and generation. In this work, we investigate how transformers with softmax attention perform in-context learning on linear classification data. We first construct a class of multi-layer transformers that can perform in-context logistic regression, with each layer exactly performing one step of normalized gradient descent on an in-context loss. Then, we show that our constructed transformer can be obtained through (i) training a single self-attention layer supervised by one-step gradient descent, and (ii) recurrently applying the trained layer to obtain a looped model. Training convergence guarantees of the self-attention layer and out-of-distribution generalization guarantees of the looped model are provided. Our results advance the theoretical understanding of ICL mechanism by showcasing how softmax transformers can effectively act as in-context learners.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。