揭示Transformer中分簇现象的数学机制，解释为何模型能长期保持稳定状态。
Emergence of meta-stable clustering in mean-field transformer models
- 将Transformer层内令牌演化建模为球面上的连续动力系统。
- 证明在大量令牌下，系统会趋向具有周期结构的准稳态解。
- 发现准稳态结构由反温度参数决定，与广义切比雪夫多项式相关。
我们将深度Transformer层中令牌的演化建模为单位球面上的连续时间流，基于Geshkovski等人（2023）提出的框架，该系统可视为均值场相互作用粒子系统。本文通过研究对应的均值场偏微分方程（PDE），该方程可解释为Wasserstein梯度流，对系统的长期行为进行数学分析，重点关注准稳态相与聚类现象的出现与持续性，这些是下一词预测等应用中的关键要素。具体而言，我们对均值场PDE在独立同分布均匀初始化附近进行摄动分析，并证明在令牌数量趋于无穷的极限下，模型将保持在具有特定结构（如周期性）的准稳态流形附近。此外，该准稳态流形的结构由模型的逆温度参数决定，其特征可通过某类缩放后的广义切比雪夫多项式的最大索引显式确定。
原文摘要 · Abstract (English)
We model the evolution of tokens within a deep stack of Transformer layers as a continuous-time flow on the unit sphere, governed by a mean-field interacting particle system, building on the framework introduced in (Geshkovski et al., 2023). Studying the corresponding mean-field Partial Differential Equation (PDE), which can be interpreted as a Wasserstein gradient flow, in this paper we provide a mathematical investigation of the long-term behavior of this system, with a particular focus on the emergence and persistence of meta-stable phases and clustering phenomena, key elements in applications like next-token prediction. More specifically, we perform a perturbative analysis of the mean-field PDE around the iid uniform initialization and prove that, in the limit of large number of tokens, the model remains close to a meta-stable manifold of solutions with a given structure (e.g., periodicity). Further, the structure characterizing the meta-stable manifold is explicitly identified, as a function of the inverse temperature parameter of the model, by the index maximizing a certain rescaling of Gegenbauer polynomials.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。