语言模型训练初期，词汇嵌入快速对齐语义语法结构。
Vocabulary embeddings organize linguistic structure early in language model training
- 通过嵌入向量几何分析，追踪词汇表示演化过程。
- 高频词和功能词的嵌入更快收敛，低频词仍受初始随机性影响。
- 揭示词频与词性在模型训练中的不同作用，适合研究模型内部机制者阅读。
大型语言模型通过多层变换输入嵌入向量的几何结构来运作。本文探究语言模型的词汇嵌入如何组织，以及这种结构在训练过程中如何演变。我们采用表示相似性分析，对两个开源模型（Pythia 12B 和 OLMo 7B）在训练全程的输入与输出嵌入进行分析，关联其几何结构与语义、句法及频率指标。关键发现：1）训练初期，词汇嵌入几何迅速与多种语义和句法特征高度相关；2）高频词与功能词（如“the”、“of”）的嵌入比词汇词和低频词更快达到最终向量，后者仍保留初始随机化偏差。这些结果揭示了输入嵌入如何动态围绕语言结构组织，明确了词频与词性的不同作用，为理解词汇几何演化如何促进能力提升提供了新视角。
原文摘要 · Abstract (English)
Large language models (LLMs) work by manipulating the geometry of input embedding vectors over multiple layers. Here, we ask: how are the input vocabulary representations of language models structured, and how and when does this structure evolve over training? To answer this question, we use representational similarity analysis, running a suite of experiments that correlate the geometric structure of the input embeddings and output embeddings of two open-source models (Pythia 12B and OLMo 7B) with semantic, syntactic, and frequency-based metrics over the course of training. Our key findings are as follows: 1) During training, the vocabulary embedding geometry quickly converges to high correlations with a suite of semantic and syntactic features; 2) Embeddings of high-frequency and function words (e.g., "the," "of") converge to their final vectors faster than lexical and low-frequency words, which retain some alignment with the bias in their random initializations. These findings help map the dynamic trajectory by which input embeddings organize around linguistic structure, revealing distinct roles for word frequency and function. Our findings motivate a deeper study of how the evolution of vocabulary geometry may facilitate specific capability gains during model training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。