解析了变压器在中等交互下的多尺度演化，揭示了注意力机制的动态分层过程。
A multiscale analysis of mean-field transformers in the moderate interaction regime
- 将注意力头视为粒子系统，用均场理论分析其深度演化行为
- 发现三阶段动力学：快速坍缩、聚类形成、缓慢合并为单一簇
- 适用于理解大模型推理时的注意力分布演化，适合研究自注意力机制的学者
本文通过将仅编码器的Transformer模型在推理时的令牌演化建模为一个均场相互作用的粒子系统，研究其深度上的动态行为。具体而言，考虑的是中等交互情形：令牌数量N较大，且模型的逆温度参数β与N一同变化。在此情形下，系统展现出多尺度特性：快速阶段，令牌的经验测度坍缩至低维空间；中间阶段，测度进一步聚类；缓慢阶段，这些簇依次合并为单一簇。论文严格刻画了各阶段的极限动态，并证明了在该极限下的收敛性，通过模拟验证了结论。
原文摘要 · Abstract (English)
In this paper, we study the evolution of tokens through the depth of encoder-only transformer models at inference time by modeling them as a system of particles interacting in a mean-field way and studying the corresponding dynamics. More specifically, we consider this problem in the moderate interaction regime, where the number $N$ of tokens is large and the inverse temperature parameter $β$ of the model scales together with $N$. In this regime, the dynamics of the system displays a multiscale behavior: a fast phase, where the token empirical measure collapses on a low-dimensional space, an intermediate phase, where the measure further collapses into clusters, and a slow one, where such clusters sequentially merge into a single one. We provide a rigorous characterization of the limiting dynamics in each of these phases and prove convergence in the above mentioned limit, exemplifying our results with some simulations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。