用微分几何重新解释Transformer,揭示其稳定性和优化机制。
The Geometry of Semantic Space: A Continuous Geometric Framework for the Transformer Architecture

- 将Transformer的各组件映射为连续几何方程,构建统一数学框架。
- 实验证实多项预测:参数波动、注意力结构、训练相变等均符合几何规律。
- 适合研究模型稳定性、优化动力学与大模型设计的学者阅读。
我们提出一个连续几何框架,将Transformer的离散代数操作建模为语义纤维丛 $\calE = \calM \times \R^d$ 上的积分微分方程(IDE)。基于单一几何公理——令牌序列构成带规范测度格的离散1-流形——我们将现代Transformer的核心组件(RMSNorm、RoPE、Softmax Attention、FFN、残差流、SGD、权重衰减)统一转化为微分几何、测度论与随机分析的语言。该框架在熵最优传输(注意力即Schrödinger桥)与非平衡热力学(SGD作为违反细致平衡的Itô扩散)方面产生定量预测。我们在五个架构(Qwen3、LLaMA-3.1、Gemma-3、GPT-2、Mistral)上开展六部分实验,参数量覆盖124M至8B。实测可观测量与几何预测高度一致:ε^{-1/2}的利普希茨标度校准(R²=1.000)、Lie-Trotter算子分裂扭率、对称消融不稳定性证实拓扑稳定性双律、在RoPE环面上$\\\\calO(1/\\\sqrt{k})$的庞加莱循环热力学抑制、热力学上下文极限相变,以及非平衡稳态参数涡旋——两种优化器(AdamW与Pure SGD)验证排除动量干扰。结果表明,通过连续随机微分几何视角分析Transformer,可为大模型的稳定性边界、上下文容量与优化动态提供可预测的描述语言。
原文摘要 · Abstract (English)
We present a continuous geometric framework that models the discrete algebraic operations of the Transformer architecture as an integro-differential equation (IDE) on a semantic fiber bundle $\calE = \calM \times \R^d$. Beginning from a single geometric axiom -- that the token sequence forms a discrete $1$-manifold equipped with a canonical measure lattice -- we translate every core component of the modern Transformer (RMSNorm, RoPE, Softmax Attention, FFN, Residual Stream, SGD, Weight Decay) into a cohesive vocabulary of differential geometry, measure theory, and stochastic calculus. The resulting framework yields quantitative predictions spanning entropic optimal transport (Attention as a Schrödinger bridge) and non-equilibrium thermodynamics (SGD as Itô diffusion violating detailed balance). We conduct a six-part experimental campaign across five architectures (Qwen3, LLaMA\nobreakdash-3.1, Gemma\nobreakdash-3, GPT-2, Mistral) spanning $124$M to $8$B parameters. The empirical observables are quantitatively consistent with the geometric predictions: the $ε^{-1/2}$ Lipschitz scaling calibration at machine precision ($R^2 = 1.000$), the Lie--Trotter operator-splitting torsion, the symmetric ablation instability confirming the Dual-Law of Topological Stability, the $\calO(1/\sqrt{k})$ thermodynamic suppression of Poincaré recurrence on the RoPE torus, the thermodynamic context-limit phase transition, and the Non-Equilibrium Steady State parameter vortex -- verified across two optimizers (AdamW and Pure SGD) to exclude momentum artifacts. The results demonstrate that analyzing Transformers through the lens of continuous stochastic differential geometry provides a predictive descriptive vocabulary for the stability limits, context bounds, and optimization dynamics of Large Language Models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。