Transformer 的注意力机制本质是普通最小二乘法的特殊形式。
Ordinary Least Squares is a Special Case of Transformer
- 通过协方差矩阵谱分解，构造参数使注意力等价于 OLS 闭式解。
- 单次前向传播即可完成回归，无需迭代优化。
- 揭示线性 Transformer 中慢速与快速记忆的解耦机制，适合理解模型原理者阅读。
Transformer 架构的统计本质长期未明：它是通用近似器，还是经典计算算法的神经网络版本？通过严格的代数证明，我们表明后者更准确描述了 Transformer 的基本性质：普通最小二乘法（OLS）是单层线性 Transformer 的特例。基于经验协方差矩阵的谱分解，我们构建了一种特定参数设置，使得注意力机制的前向传播在数学上等价于 OLS 的闭式投影。这意味着注意力可在一次前向传播中求解问题，而非通过迭代优化。在此原型基础上，我们进一步揭示了 Transformer 内部存在的解耦慢速与快速记忆机制。最后，讨论了从该线性原型到标准 Transformer 的演化过程，该过程促使霍普菲尔德能量函数从线性扩展至指数级记忆容量，从而在现代深度架构与经典统计推断之间建立了清晰的连续性。
原文摘要 · Abstract (English)
The statistical essence of the Transformer architecture has long remained elusive: Is it a universal approximator, or a neural network version of known computational algorithms? Through rigorous algebraic proof, we show that the latter better describes Transformer's basic nature: Ordinary Least Squares (OLS) is a special case of the single-layer Linear Transformer. Using the spectral decomposition of the empirical covariance matrix, we construct a specific parameter setting where the attention mechanism's forward pass becomes mathematically equivalent to the OLS closed-form projection. This means attention can solve the problem in one forward pass, not by iterating. Building upon this prototypical case, we further uncover a decoupled slow and fast memory mechanism within Transformers. Finally, the evolution from our established linear prototype to standard Transformers is discussed. This progression facilitates the transition of the Hopfield energy function from linear to exponential memory capacity, thereby establishing a clear continuity between modern deep architectures and classical statistical inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。