用优化理论设计加速版Transformer，性能优于标准模型。
YuriiFormer: A Suite of Nesterov-Accelerated Transformers
- 将自注意力与MLP视为优化迭代中的梯度步，构建可解释框架。
- 引入奈斯特罗夫加速机制，在TinyStories和OpenWebText上超越nanoGPT。
- 适合关注模型架构设计原理的算法研究者和深度学习工程师。
我们提出一种变分框架，将Transformer层视为作用于标记嵌入的优化算法迭代过程。在此视角下，自注意力实现交互能量的梯度步，MLP层对应势能的梯度更新。标准GPT类Transformer可看作对复合目标函数进行的朴素梯度下降，通过李-特罗特分裂在两个能量泛函间实现。该视角使基于经典优化思想的结构设计成为可能。作为概念验证，我们引入一种保留相同注意力与MLP算子的奈斯特罗夫风格加速Transformer。其在TinyStories和OpenWebText数据集上持续优于nanoGPT基线，表明优化理论洞见可转化为实际性能提升。
原文摘要 · Abstract (English)
We propose a variational framework that interprets transformer layers as iterations of an optimization algorithm acting on token embeddings. In this view, self-attention implements a gradient step of an interaction energy, while MLP layers correspond to gradient updates of a potential energy. Standard GPT-style transformers emerge as vanilla gradient descent on the resulting composite objective, implemented via Lie--Trotter splitting between these two energy functionals. This perspective enables principled architectural design using classical optimization ideas. As a proof of concept, we introduce a Nesterov-style accelerated transformer that preserves the same attention and MLP oracles. The resulting architecture consistently outperforms a nanoGPT baseline on TinyStories and OpenWebText, demonstrating that optimization-theoretic insights can translate into practical gains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。