揭示Transformer注意力的集体行为与聚类机制。
The Mean-Field Dynamics of Transformers
- 将注意力建模为粒子系统，分析其连续极限
- 发现长期稳定后令牌自动聚类，形成多簇结构
- 揭示归一化与长序列注意力的相变规律
我们构建了一个数学框架，将Transformer注意力视为相互作用的粒子系统，并研究其连续（均值场）极限。通过理想化球面上的注意力，我们将Transformer动态与Wasserstein梯度流、同步模型（Kuramoto）及均值漂移聚类相联系。核心结果是全局聚类现象：在经历长时间的亚稳态后，令牌会渐进地聚集为多个簇。我们进一步通过可解析的等角简化得到精确的聚类速率，展示了常用归一化方式如何改变收缩速度，并识别出长上下文注意力的相变点。结果揭示了表征坍塌的机制，以及维持深层注意力架构中丰富多簇结构的区域。
原文摘要 · Abstract (English)
We develop a mathematical framework that interprets Transformer attention as an interacting particle system and studies its continuum (mean-field) limits. By idealizing attention on the sphere, we connect Transformer dynamics to Wasserstein gradient flows, synchronization models (Kuramoto), and mean-shift clustering. Central to our results is a global clustering phenomenon whereby tokens cluster asymptotically after long metastable states where they are arranged into multiple clusters. We further analyze a tractable equiangular reduction to obtain exact clustering rates, show how commonly used normalization schemes alter contraction speeds, and identify a phase transition for long-context attention. The results highlight both the mechanisms that drive representation collapse and the regimes that preserve expressive, multi-cluster structure in deep attention architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。