发现嵌入向量均值模长越小，其方差越大，揭示Transformer层间特征分布规律。
Norm of Mean Contextualized Embeddings Determines their Variance
- 通过统计方差公式分析嵌入分布，提出高效序列计算方法
- 发现均值模长与方差存在强负相关关系，随层数加深方差分布更分散
- 适用于研究模型内部表示特性，尤其关注嵌入空间结构的读者
上下文嵌入会因语境变化而波动，形成嵌入空间中的分布。本文聚焦于均值嵌入的模长与嵌入方差，首先验证其符合统计学中方差公式，并提出高效序列计算方法。通过对多个Transformer模型中间层嵌入的观察，发现均值嵌入越接近原点，方差越大，二者存在显著权衡关系，可能由层归一化机制导致。当将词嵌入视为簇时，理论证明整体方差可分解为簇内与簇间方差。实验表明，随着模型层数加深，嵌入远离原点，簇间方差相对减小，簇内方差相对增大，该结果与嵌入空间各层各向异性研究一致。
原文摘要 · Abstract (English)
Contextualized embeddings vary by context, even for the same token, and form a distribution in the embedding space. To analyze this distribution, we focus on the norm of the mean embedding and the variance of the embeddings. In this study, we first demonstrate that these values follow the well-known formula for variance in statistics and provide an efficient sequential computation method. Then, by observing embeddings from intermediate layers of several Transformer models, we found a strong trade-off relationship between the norm and the variance: as the mean embedding becomes closer to the origin, the variance increases. This trade-off is likely influenced by the layer normalization mechanism used in Transformer models. Furthermore, when the sets of token embeddings are treated as clusters, we show that the variance of the entire embedding set can theoretically be decomposed into the within-cluster variance and the between-cluster variance. We found experimentally that as the layers of Transformer models deepen, the embeddings move farther from the origin, the between-cluster variance relatively decreases, and the within-cluster variance relatively increases. These results are consistent with existing studies on the anisotropy of the embedding spaces across layers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。