发现Transformer模型的词嵌入空间蕴含丰富语义信息，支持词汇存储假说。
Word Meanings in Transformer Language Models
- 对RoBERTa-base的词嵌入进行200聚类，探索语义结构
- 聚类结果显著关联五种心理语言学指标（如具体性、情感极性）
- 结果挑战了'语义消解论'，支持模型内存在词汇意义表征
本文研究Transformer语言模型中词义的表征方式，重点考察模型是否具有类似词汇存储的机制——即每个词都有包含语义信息的独立条目。为此，我们提取了RoBERTa-base的词嵌入空间，并使用k-means聚类成200个簇。第一项研究通过人工检查聚类结果，评估其对语义信息的敏感性；第二项研究检验这些簇是否与五项心理语言学特征相关：情感极性（valence）、具体性（concreteness）、象似性（iconicity）、禁忌性（taboo）和习得年龄（age of acquisition）。总体结果显示，词嵌入空间中编码了广泛的语义信息，有力排除了某些关于变压器语言模型处理语义信息的‘意义消解论’假设。
原文摘要 · Abstract (English)
We investigate how word meanings are represented in the transformer language models. Specifically, we focus on whether transformer models employ something analogous to a lexical store - where each word has an entry that contains semantic information. To do this, we extracted the token embedding space of RoBERTa-base and k-means clustered it into 200 clusters. In our first study, we then manually inspected the resultant clusters to consider whether they are sensitive to semantic information. In our second study, we tested whether the clusters are sensitive to five psycholinguistic measures: valence, concreteness, iconicity, taboo, and age of acquisition. Overall, our findings were very positive - there is a wide variety of semantic information encoded within the token embedding space. This serves to rule out certain "meaning eliminativist" hypotheses about how transformer LLMs process semantic information.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。