解释词向量中线性类比关系的成因，揭示其来自语义属性的隐式结构。
On the Emergence of Linear Analogies in Word Embeddings
- 基于语义属性生成模型，从理论上重现词向量中的线性类比
- 证明类比结构在主特征向量中即出现，并随维度增加趋于饱和
- 适用于理解词向量内在机制，适合研究语言表示的学者
Word2Vec 和 GloVe 等模型通过文本中词对 $i,j$ 的共现概率 $P(i,j)$ 构建词向量 $W_i$，不仅使语义相近的词聚类，还展现出显著的线性类比结构，如 $W_{ ext{king}} - W_{ ext{man}} + W_{ ext{woman}} riangleq W_{ ext{queen}}$。该现象的理论来源尚不明确。先前观察发现：（i）类比结构已存在于矩阵 $M(i,j) = P(i,j)/P(i)P(j)$ 的前几个特征向量中；（ii）随着更多特征向量被纳入，类比性能先增强后饱和；（iii）使用 $\ ext{log} M(i,j)$ 比 $M(i,j)$ 更能增强类比；（iv）即使移除特定类比词对（如 king-queen, man-woman），类比仍存在。本文提出一个生成模型，其中词由二值语义属性定义，共现概率由属性交互决定。该模型可解析推导出线性类比结构，自然解释上述四点现象，并提供对每个嵌入维度作用的细粒度洞察。模型对多种噪声具有鲁棒性，与 Wikipedia 和 Mikolov 等人提出的类比基准数据统计高度吻合。
原文摘要 · Abstract (English)
Models such as Word2Vec and GloVe construct word embeddings based on the co-occurrence probability $P(i,j)$ of words $i$ and $j$ in text corpora. The resulting vectors $W_i$ not only group semantically similar words but also exhibit a striking linear analogy structure -- for example, $W_{\text{king}} - W_{\text{man}} + W_{\text{woman}} \approx W_{\text{queen}}$ -- whose theoretical origin remains unclear. Previous observations indicate that this analogy structure: (i) already emerges in the top eigenvectors of the matrix $M(i,j) = P(i,j)/P(i)P(j)$, (ii) strengthens and then saturates as more eigenvectors of $M (i, j)$, which controls the dimension of the embeddings, are included, (iii) is enhanced when using $\log M(i,j)$ rather than $M(i,j)$, and (iv) persists even when all word pairs involved in a specific analogy relation (e.g., king-queen, man-woman) are removed from the corpus. To explain these phenomena, we introduce a theoretical generative model in which words are defined by binary semantic attributes, and co-occurrence probabilities are derived from attribute-based interactions. This model analytically reproduces the emergence of linear analogy structure and naturally accounts for properties (i)-(iv). It can be viewed as giving fine-grained resolution into the role of each additional embedding dimension. It is robust to various forms of noise and agrees well with co-occurrence statistics measured on Wikipedia and the analogy benchmark introduced by Mikolov et al.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。