对比不同规模大模型的词义网络，发现越大模型词义结构越复杂。
From the New World of Word Embeddings: A Comparative Study of Small-World Lexico-Semantic Networks in LLMs
- 用大模型输入嵌入构建词义网络，分析其全局与局部结构
- 大模型词义网络具小世界特性，参数越多路径越长、结构越复杂
- 适用于研究模型语义表征，尤其关注大模型内部机制
词义网络将词语视为节点，语义相关性作为边。传统上这类网络基于编码器模型或静态向量构建，而解码器仅有的大语言模型（LLM）嵌入尚未被充分探索。与编码器不同，LLM通过预测下一个词训练，不直接编码当前词意义。本文从不同参数规模的LLM输入嵌入构建词义网络，并比较其全局与局部结构。结果表明，这些网络具有小世界特征：高聚类性与短路径长度。更大的LLM生成更复杂的网络，小世界效应减弱，路径变长，反映更丰富的语义结构与关系。我们通过常见概念对、基于WordNet的结构化词汇关系及跨语言语义网络对结果进行了定性验证。
原文摘要 · Abstract (English)
Lexico-semantic networks represent words as nodes and their semantic relatedness as edges. While such networks are traditionally constructed using embeddings from encoder-based models or static vectors, embeddings from decoder-only large language models (LLMs) remain underexplored. Unlike encoder models, LLMs are trained with a next-token prediction objective, which does not directly encode the meaning of the current token. In this paper, we construct lexico-semantic networks from the input embeddings of LLMs with varying parameter scales and conduct a comparative analysis of their global and local structures. Our results show that these networks exhibit small-world properties, characterized by high clustering and short path lengths. Moreover, larger LLMs yield more intricate networks with less small-world effects and longer paths, reflecting richer semantic structures and relations. We further validate our approach through analyses of common conceptual pairs, structured lexical relations derived from WordNet, and a cross-lingual semantic network for qualitative words.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。