发现Transformer层与幂法的数学相似性，揭示其隐含的主特征向量对齐机制。
Analogies between Transformer Layers and Power Method
- 将Transformer层操作类比为幂法迭代步骤
- 令牌趋向于对齐输出与值矩阵乘积的主特征向量
- 可据此设计方法引导输出方向，适用于模型可控生成
本文揭示了Transformer层中的投影与层归一化操作（忽略前馈网络）与幂法步骤之间的类比关系。根据这一类比，我们证明通过一层后，令牌会趋向于对齐该层输出权重矩阵与值权重矩阵乘积的主特征向量。在权重共享的Transformer中（即所有层权重相同），这种对齐在实验中尤为明显，并可通过分析严格证明。该类比还启发了一种新方法，可将Transformer输出引导至令牌空间中的任意期望方向。
原文摘要 · Abstract (English)
In the paper we show that there is an analogy between the operations occurring in a layer of a transformer (projections and layer normalizations, disregarding the feedforward neural network) and a step in the power method. Coherently with this analogy, we show that passing through a layer the tokens tend to be tilted towards the principal eigenvector of a matrix which is the product of the output and value weight matrices of that layer. In the special case of a transformer with shared weights (i.e., in which all layers have identical weights) then the alignment with this principal eigenvector is particularly evident empirically, and can also be shown analytically. The analogy also suggests a method to steer the output of the transformer towards an arbitrary desired direction in token space.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。