揭示了变压器模型中词元随深度演化呈现指数级聚类现象。
Quantitative Clustering in Mean-Field Transformer Models
- 将变压器模型视为粒子系统，分析词元演化规律。
- 证明在合理参数下，初始状态以明确速率快速聚成单一峰值。
- 适用于研究大模型内部表示动力学的学者。
深度变压器模型中词元的演化可建模为一个相互作用的粒子系统，已有研究表明其表现出类似Kuramoto模型同步现象的渐近聚类行为。本文研究均值场变压器模型的长期聚类特性。在对变压器模型参数施加适当假设的前提下，我们证明任意合适的正则初始状态将以明确的定量收敛速率，指数级快速同步至狄拉克点质量。
原文摘要 · Abstract (English)
The evolution of tokens through deep transformer models can be modeled as an interacting particle system that has been shown to exhibit an asymptotic clustering behavior akin to the synchronization phenomenon in Kuramoto models. In this work, we investigate the long-time clustering of mean-field transformer models. More precisely, under suitable assumptions on the transformer model parameters, we establish that any suitably regular mean-field initialization synchronizes exponentially fast to a Dirac point mass, with explicit quantitative convergence rates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。