研究Transformer中感知机模块如何影响注意力的全局分布特性。
Perceptrons and localization of attention's mean-field landscape
- 将Transformer前向传播建模为球面上的粒子系统，用变分梯度流分析
- 发现关键点通常集中在球面特定子集上，具有局域化特征
- 适合对注意力机制几何性质感兴趣的理论研究者
Transformer的前向传播可被看作单位球面上的相互作用粒子系统：时间对应层数，粒子对应标记嵌入，单位球面则理想化了层归一化。在某些权重设置下，该系统可视为显式能量的梯度流，借助Wasserstein梯度流可定义无限上下文长度（均场）极限。本文研究在此设定下感知机块的影响，证明临界点通常是原子性的，且局域于球面的子集上。
原文摘要 · Abstract (English)
The forward pass of a Transformer can be seen as an interacting particle system on the unit sphere: time plays the role of layers, particles that of token embeddings, and the unit sphere idealizes layer normalization. In some weight settings the system can even be seen as a gradient flow for an explicit energy, and one can make sense of the infinite context length (mean-field) limit thanks to Wasserstein gradient flows. In this paper we study the effect of the perceptron block in this setting, and show that critical points are generically atomic and localized on subsets of the sphere.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。