随机二次型在球面上实现由共噪声引发的同步，解释了注意力机制外的聚类现象。
Random Quadratic Form on a Sphere: Synchronization by Common Noise
- 构建随机二次型模型，模拟线性层在变换器中的动力学行为。
- 发现即使无自注意力，各变量仍会因共噪声产生同步聚类。
- 为深层变换器中令牌聚类提供非注意力机制的新解释，适合研究神经网络动态者。
我们提出随机二次型（RQF）：一种定义在球面上的随机微分方程，形式上对应于随机二次泛函的梯度流。单点动力学表现为布朗运动，无偏好方向；但两点动力学表现出非平凡的同步行为。本文通过研究系统的不变测度与随机吸引子，从分布与路径两个层面刻画了RQF的同步特性。该模型源于对变换器中线性层作用的研究，揭示了简化模型中由共噪声引发的同步现象。特别地，我们提供了独立于自注意力机制的深层变换器中令牌聚类行为的新解释——即使没有自注意力，令牌仍会聚类。
原文摘要 · Abstract (English)
We introduce the Random Quadratic Form (RQF): a stochastic differential equation which formally corresponds to the gradient flow of a random quadratic functional on a sphere. While the one-point dynamics of the system is a Brownian motion and thus has no preferred direction, the two-point motion exhibits nontrivial synchronizing behaviour. In this work we study synchronization of the RQF, namely we give both distributional and path-wise characterizations of the solutions by studying invariant measures and random attractors of the system. The RQF model is motivated by the study of the role of linear layers in transformers and illustrates the synchronization by common noise phenomena arising in the simplified models of transformers. In particular, we provide an alternative (independent of self-attention) explanation of the clustering behaviour in deep transformers and show that tokens cluster even in the absence of the self-attention mechanism.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。