揭示了Transformer深度增加时所有标记最终趋于聚集的数学本质。
The Asymptotic Behavior of Attention in Transformers
- 用控制理论分析Transformer动态，建立深度与标记收敛的关系
- 证明随着层数加深，所有标记会收敛到单一簇
- 适用于大语言模型设计，尤其关注深度扩展的局限性
Transformer架构已成为现代大语言模型（LLMs）的基础，但其理论性质仍不明确。尽管增大模型规模和深度是常见优化策略，但多项研究指出，增加层数带来的收益递减，甚至可能引发模型坍缩——即所有标记趋于同一簇，削弱生成多样性。本文基于变换器动态的微分方程模型，证明随着深度增加，所有标记将渐近收敛至一个簇。技术上运用了控制理论工具，包括流形上的共识动力学与输入-状态稳定性（ISS）。进一步地，利用自回归模型结构，拓展了理论保证的适用范围。
原文摘要 · Abstract (English)
The transformer architecture has become the foundation of modern Large Language Models (LLMs), yet its theoretical properties are still not well understood. As with classic neural networks, a common approach to improve these models is to increase their size and depth. However, such strategies may be suboptimal, as several works have shown that adding more layers yields increasingly diminishing returns. More importantly, prior studies have shown that increasing depth may lead to model collapse, i.e., all the tokens converge to a single cluster, undermining the ability of LLMs to generate diverse outputs. Building on differential equation models for the transformer dynamics, we prove that all the tokens in a transformer asymptotically converge to a cluster as depth increases. At the technical level we leverage tools from control theory, including consensus dynamics on manifolds and input-to-state stability (ISS). We then extend our analysis to autoregressive models, exploiting their structure to further generalize the theoretical guarantees.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。