从最大似然与最大熵推导Transformer的缩放点积函数
Deriving the Scaled-Dot-Function via Maximum Likelihood Estimation and Maximum Entropy Approach
- 用高斯分布建模键、值、查询向量序列，通过最大似然估计求解值向量
- 发现缩放点积函数可由最大熵模型中的特征函数自然导出
- 为Transformer注意力机制提供统计学习理论解释，适合研究者参考
本文提出一种基于最大似然估计的方法，用于确定Transformer模型中的值向量。我们将值向量、键向量和查询向量的序列建模为高斯分布序列，其中每个高斯分布的方差依赖于时间步、对应的键向量和查询向量，均值则依赖于时间步和对应的值向量。该分析为Transformer架构中使用的缩放点积函数或Softmax函数提供了新解释。另一个解释受[4]启发，源自自然语言处理中的最大熵方法[5]：查询向量与键向量共同构成最大熵模型的特征函数。
原文摘要 · Abstract (English)
In this paper, we present a maximum likelihood estimation approach to determine the value vector in transformer models. We model the sequence of value vectors, key vectors, and the query vector as a sequence of Gaussian distributions. The variance in each Gaussian distribution depends on the time step, the corresponding key vector, and the query vector. The mean value in each Gaussian distribution depends on the time step, and the corresponding value vector. This analysis may offer a new explanation of the scaled-dot-product function or softmax function used in transformer architectures [1]. Another explanation, inspired by [4], is based on the maximum entropy approach in natural language processing [5]. In this approach, a query vector and key vectors are used to derive the feature functions for the maximum entropy model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。