揭示深度Transformer中噪声如何引发同步与能量衰减
Stochastic Scaling Limits and Synchronization by Noise in Deep Transformer Models
- 将离散Transformer层演化解析为连续随机粒子系统
- 证明噪声足够强时，交互能量指数级衰减
- 适用于研究深度模型动态的理论学者
我们证明了具有多层感知机（MLP）模块的有限深度、有限宽度Transformer模型中，各层令牌的演化路径几乎必然收敛到一个连续时间的随机相互作用粒子系统。同时，我们识别出描述令牌分布演化的随机偏微分方程，并在令牌数量大时证明了混沌传播。所建立的界是定量的，且所考虑的极限可交换。进一步证明，在共同噪声相对于确定性自注意力漂移足够强的情况下，极限随机模型表现出噪声驱动的同步现象，并平均意义上实现交互能量的指数衰减。最后，我们刻画了满足该条件的激活函数形式。
原文摘要 · Abstract (English)
We prove pathwise convergence of the layerwise evolution of tokens in a finite-depth, finite-width transformer model with MultiLayer Perceptron (MLP) blocks to a continuous-time stochastic interacting particle system. We also identify the stochastic partial differential equation describing the evolution of the tokens' distribution in this limit and prove propagation of chaos when the number of such tokens is large. The bounds we establish are quantitative and the limits we consider commute. We further prove that the limiting stochastic model displays synchronization by noise and establish exponential dissipation of the interaction energy on average, provided that the common noise is sufficiently coercive relative to the deterministic self-attention drift. We finally characterize the activation functions satisfying the former condition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。