揭示了变压器层中语法角色如何改变词向量的几何结构
The Changing Geometry of Grammar: Dimensionality and Neighborhood Reorganization across Transformer Layers
- 分析各层词向量的内在维度变化，发现封闭类词扩展早、收缩快
- 发现词邻域结构变化是维度演变的关键，反映句内关系动态
- 仅用几何特征就能预测词性，适合语言模型可解释性研究者
Transformer 的表示在高维空间中形成轨迹，随层推进动态演化。这些数据常聚集于低维子流形，其压缩程度可用内在维度（ID）量化，即无显著信息损失所需的最小独立变量数。本文探究词性（PoS）是否塑造该流形的局部几何：(1) 分析层间 ID 变化，发现封闭类词（如介词）比开放类词（如名词）更早扩展、更早坍缩；(2) 证明其扩张与收缩由词间邻域结构变化驱动，反映句法关系演变；(3) 比较编码器（ModernBERT、bigbird-roberta-large）与解码器（gemma-2-2B、Llama-3.2-3B），发现两类模型跨层演化模式不同，与其上下文整合方式一致；(4) 仅凭几何特征即可恢复词性标签，并用于下游分类任务中分析各类词性语义在层间的演化路径。
原文摘要 · Abstract (English)
Transformer representations describe trajectories through high-dimensional vector spaces, which are shaped dynamically as tokens incorporate relational context across layers. Such data tend to concentrate on lower-dimensional sub-manifolds, a form of compression quantified by the Intrinsic Dimensionality (ID), the minimum number of independent variables needed to represent them without significant information loss. In this work, we ask whether the grammatical role of tokens, as marked by their part-of-speech (PoS) tag, shapes the local geometry of this manifold. To this end: (1) We investigate the layer-wise evolution of ID, finding that closed-class items expand earlier and collapse sooner than open-class ones; (2) We show its expansion and contraction to be explained by changes in the neighborhood structure, and hence in the relations between words within a sentence; (3) We compare encoders (ModernBERT, bigbird-roberta-large) and decoders (gemma-2-2B, Llama-3.2-3B), finding that the two families evolve differently across layers, consistently with how each integrates context;(4) We show that geometric features alone recover a token's grammatical role, and use them to interpret how the semantic content of each PoS evolves across layers in a downstream classification task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。