将Transformer层更新视为连续动力系统，揭示其稳定与表达机制
Flowing Through Layers: A Continuous Dynamical Systems Perspective on Transformers
- 用欧拉法将离散层更新看作连续微分方程的离散化
- 层数增多时，令牌表征统一收敛到唯一ODE解
- 负单边利普希茨条件使扰动指数衰减，解释模型稳定性
我们证明，标准Transformer层的离散更新规则可自然地被理解为连续动力系统的前向欧拉离散化。我们的Transformer流近似定理表明，在标准利普希茨连续性假设下，当层数增加时，令牌表示以一致方式收敛到一个微分方程的唯一解。此外，若底层映射满足具有负常数的单边利普希茨条件，则系统动态具有收缩性，导致扰动在各层间呈指数衰减。这些发现不仅澄清了Transformer模型的实证稳定性和表达能力，还将其更新机制与更广泛的迭代推理框架联系起来，为加速收敛和受动力系统理论启发的架构创新提供了新思路。
原文摘要 · Abstract (English)
We show that the standard discrete update rule of transformer layers can be naturally interpreted as a forward Euler discretization of a continuous dynamical system. Our Transformer Flow Approximation Theorem demonstrates that, under standard Lipschitz continuity assumptions, token representations converge uniformly to the unique solution of an ODE as the number of layers grows. Moreover, if the underlying mapping satisfies a one-sided Lipschitz condition with a negative constant, the resulting dynamics are contractive, causing perturbations to decay exponentially across layers. Beyond clarifying the empirical stability and expressivity of transformer models, these insights link transformer updates to a broader iterative reasoning framework, suggesting new avenues for accelerated convergence and architectural innovations inspired by dynamical systems theory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。