解析Word2Vec训练动态，发现模型逐层学习可解释的主题空间。
Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models
- 通过泰勒展开解析训练轨迹，仅用语料统计与超参推导出完整解
- 模型逐个学习正交线性子空间，提升嵌入的有效秩直至容量饱和
- 顶层子空间对应可解释主题，支持通过向量加法完成类比推理
自监督词嵌入算法如Word2Vec为语言建模表征学习提供了极简设置。我们对Word2Vec损失在原点处进行四阶泰勒展开，发现其训练动态与下游任务表现与原始Word2Vec高度一致。主要贡献在于：从语料统计与训练超参数出发，解析求解了梯度流训练动态与最终词嵌入。解显示，模型逐次学习正交线性子空间，每次提升嵌入的有效秩，直至模型容量饱和。在Wikipedia上训练时，前几个主子空间分别代表可解释的主题级概念。最后，我们将理论应用于分析抽象语义概念的生成机制；这些线性表示可用于通过向量加法完成类比任务。
原文摘要 · Abstract (English)
Self-supervised word embedding algorithms such as word2vec provide a minimal setting for studying representation learning in language modeling. We examine the quartic Taylor approximation of the word2vec loss around the origin, and we show that both the resulting training dynamics and the final performance on downstream tasks are empirically very similar to those of word2vec. Our main contribution is to analytically solve for both the gradient flow training dynamics and the final word embeddings in terms of only the corpus statistics and training hyperparameters. The solutions reveal that these models learn orthogonal linear subspaces one at a time, each one incrementing the effective rank of the embeddings until model capacity is saturated. Training on Wikipedia, we find that each of the top linear subspaces represents an interpretable topic-level concept. Finally, we apply our theory to describe how linear representations of more abstract semantic concepts emerge during training; these can be used to complete analogies via vector addition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。