揭示Transformer自注意力动态的多重稳定性机制
Multistability of Self-Attention Dynamics in Transformers
- 将自注意力建模为多智能体系统，类比于Oja流
- 发现三种共存的稳定平衡态：共识、双共识与聚类
- 稳定状态常对齐于值矩阵主特征向量，适用于理解模型行为
在机器学习中，自注意力动力学是一种连续时间的多智能体类模型，用于描述Transformer注意力机制。本文表明该动力学与多智能体版本的Oja流相关，后者可计算矩阵的主特征向量，对应于Transformer中的值矩阵。我们对单头自注意力系统中的平衡点进行分类，分为四类：共识、双共识、聚类和多边形平衡态。前三种类型的多个渐近稳定平衡态常在自注意力动力学中共存。有趣的是，前两类平衡态始终与值矩阵的特征向量对齐，通常但不总是与主特征向量对齐。
原文摘要 · Abstract (English)
In machine learning, a self-attention dynamics is a continuous-time multiagent-like model of the attention mechanisms of transformers. In this paper we show that such dynamics is related to a multiagent version of the Oja flow, a dynamical system that computes the principal eigenvector of a matrix corresponding for transformers to the value matrix. We classify the equilibria of the ``single-head'' self-attention system into four classes: consensus, bipartite consensus, clustering and polygonal equilibria. Multiple asymptotically stable equilibria from the first three classes often coexist in the self-attention dynamics. Interestingly, equilibria from the first two classes are always aligned with the eigenvectors of the value matrix, often but not exclusively with the principal eigenvector.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。