用Transformer实现隐式梯度下降，让模型自己训练自己。
In-Context Deep Learning via Transformer Models
- 构建多层Transformer，通过上下文学习模拟深度网络的梯度更新
- 理论证明可逼近任意误差，且在3-6层网络上性能媲美直接训练
- 适用于想理解模型自训练机制的研究者和开发者
我们研究了Transformer通过上下文学习（ICL）模拟深度模型训练过程的能力，即“上下文深度学习”。核心贡献是首次提供了一个正例：利用Transformer以隐式方式通过ICL执行梯度下降，训练深度神经网络。具体而言，我们构造了一个(2N+4)L层的Transformer，能够通过ICL模拟一个N层ReLU网络的L步梯度下降过程，并给出了逼近误差的理论保证及ICL梯度下降的收敛性证明。此外，分析还扩展到更实用的Softmax-based Transformer设置。在合成数据集上对3层、4层和6层神经网络进行了验证，结果表明ICL性能与直接训练相当。
原文摘要 · Abstract (English)
We investigate the transformer's capability to simulate the training process of deep models via in-context learning (ICL), i.e., in-context deep learning. Our key contribution is providing a positive example of using a transformer to train a deep neural network by gradient descent in an implicit fashion via ICL. Specifically, we provide an explicit construction of a $(2N+4)L$-layer transformer capable of simulating $L$ gradient descent steps of an $N$-layer ReLU network through ICL. We also give the theoretical guarantees for the approximation within any given error and the convergence of the ICL gradient descent. Additionally, we extend our analysis to the more practical setting using Softmax-based transformers. We validate our findings on synthetic datasets for 3-layer, 4-layer, and 6-layer neural networks. The results show that ICL performance matches that of direct training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。