用广义相对论类比Transformer,揭示注意力导致的嵌入空间弯曲。
The Curved Spacetime of Transformer Architectures
- 将注意力视为平行移动的离散连接,层间演化如时空曲率中的路径。
- 实验显示嵌入轨迹存在可测量的弯曲,且与语义一致。
- 适合对模型内部机制感兴趣的研究人员阅读。
我们提出一个几何框架来理解基于Transformer的语言模型,将其与广义相对论进行类比。查询和键向量在表示空间中诱导出一个有效度量,而注意力则作为离散联络,实现值向量在标记间的平行传输。堆叠的层构成离散的时间片,使标记表示在此弯曲流形上演化,反向传播则扮演最小作用原理的角色,塑造参数空间中最小损失的轨迹。若该类比成立,标记嵌入在特征空间中不应沿直线移动;相反,其逐层步长应因嵌入空间曲率的相互作用而发生弯曲和重定向。为验证这一预测,我们设计了实验以揭示曲率的存在及其后果:(i) 可视化整段文本的曲率景观,揭示不同标记与层间局部转向角的变化;(ii) 通过模拟证明尖锐/平缓角度的过度计数以及更长的长度-弦比无法由维度或随机性解释;(iii) 受爱因斯坦日食实验启发,通过受控上下文编辑探测偏转,证实注意力引起的嵌入轨迹存在可测量、语义一致的弯曲,从而验证曲率效应。
原文摘要 · Abstract (English)
We present a geometric framework for understanding Transformer-based language models, drawing an explicit analogy to General Relativity. Queries and keys induce an effective metric on representation space, and attention acts as a discrete connection that implements parallel transport of value vectors across tokens. Stacked layers provide discrete time-slices through which token representations evolve on this curved manifold, while backpropagation plays the role of a least-action principle that shapes loss-minimizing trajectories in parameter space. If this analogy is correct, token embeddings should not traverse straight paths in feature space; instead, their layer-wise steps should bend and reorient as interactions mediated by embedding space curvature. To test this prediction, we design experiments that expose both the presence and the consequences of curvature: (i) we visualize a curvature landscape for a full paragraph, revealing how local turning angles vary across tokens and layers; (ii) we show through simulations that excess counts of sharp/flat angles and longer length-to-chord ratios are not explainable by dimensionality or chance; and (iii) inspired by Einstein's eclipse experiment, we probe deflection under controlled context edits, demonstrating measurable, meaning-consistent bends in embedding trajectories that confirm attention-induced curvature.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。