用最优控制理论推导出类似Transformer的推理结构，揭示其内在机制。
Transformer-like Inference from Optimal Control
- 从最优控制出发，推导出与Transformer层结构一致的推理算法
- 当嵌入维度不足时，Transformer会隐式利用非马尔可夫结构
- 为Transformer的层结构提供了理论依据，适合模型原理研究者
仅解码器的Transformer通过历史观测序列计算下一个词的概率。本文从第一性原理出发，推导出解决相同预测问题的推理架构，并发现其结果恰好对应于最优控制理论中的变换结构。研究针对两类模型：一类是直接源于Transformer的离散值过程非线性模型；另一类是线性高斯模型，作为可解析的基准。在两类模型中，预测目标均被重述为最优控制问题，其解给出一种显式的推理算法——对偶滤波器，其层结构与仅解码器的Transformer完全对应。数值实验对比了最优控制结果与训练后Transformer的注意力权重，发现当嵌入维度不足时，Transformer会隐式利用非马尔可夫结构。
原文摘要 · Abstract (English)
Decoder-only transformers compute the conditional probability of the next token from a sequence of past observations. This paper derives, from first principles, inference architectures that solve the same prediction problem - and in doing so, recovers transformer-like layer operations as a consequence of optimal control theory. The framework is developed for two model classes: a nonlinear model of discrete-valued processes, directly motivated by the transformer, and a linear Gaussian model as a tractable baseline. For both model classes, the prediction objective is reformulated as an optimal control problem whose solution yields an explicit inference algorithm, the dual filter, with a layer structure that mirrors the layer structure of a decoder-only transformer. Numerical experiments provide a comparison of the optimal control to attention weights from a trained transformer. These experiments reveal that when the embedding dimension is insufficient, the transformer implicitly exploits non-Markovian structure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。