用线性注意力Transformer实现线性回归闭式解，无需梯度下降。
In-context learning of closed form solution to simple linear regression task using transformer with linear self-attention
- 采用线性自注意力结构，通过层归一化逼近闭式解。
- 在简单回归任务中，直接学习最小二乘估计，不依赖迭代优化。
- 适合关注模型内在机制与可解释性的研究者。
上下文学习是Transformer的显著特性，近年来备受关注。尽管多项研究表明Transformer能实现线性与非线性回归问题的求解，且多数方法基于梯度下降算法，但这些能力是否真正通过训练获得仍不明确。本文构建了一种具有线性自注意力的Transformer，在简单回归任务中实现上下文学习的最小二乘估计。关键在于，利用层归一化近似得到闭式（解析）解，而非依赖梯度下降的近似解。实验表明，当目标输出为最小二乘估计时，该实现主要在使用l1正则化的Transformer训练中表现良好。
原文摘要 · Abstract (English)
In-context learning is a remarkable property of transformers and has recently received a lot of interest. In many studies of in-context learning, it has been shown that transformers are capable of implementing solver for linear and non-linear regression problems, in which the most of them implement gradient descent algorithm. However, it is still unclear whether those implementations have actually been acquired through training. In this paper, we construct a transformer with linear self-attention, which in-context learns the least squares estimate in a simple regression task. The point here is that the closed form (analytical) solution is approximately obtained by using layer normalization rather than an approximate solution based on gradient descent algorithm. Then, we show an experimental example, in which our implementation is mainly used in the transformer trained with l1 regularization when the target output is the least squares estimate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。