发现大模型生成文本中词汇语义随训练出现先升后降的规律
Emergent Lexical Semantics in Neural Language Models: Testing Martin's Law on LLM-Generated Text
- 用上下文嵌入聚类定义词义，追踪模型训练过程中的语义演化
- 词汇频率与多义性相关性在第104轮达到峰值（r > 0.6），随后下降
- 小模型后期语义崩溃，大模型更稳定，提示存在最优语义窗口
我们首次系统研究了神经语言模型在训练过程中生成文本时，马丁定律——即词汇频率与多义性之间的经验关系——的表现。采用DBSCAN对上下文嵌入进行聚类以操作化词义，分析了四个Pythia模型（70M-1B参数）在30个训练检查点上的表现。结果揭示出非单调的发展轨迹：马丁定律在检查点100左右出现，于检查点104达到最高相关性（r > 0.6），随后在检查点105开始退化。较小模型（70M、160M）在后期出现灾难性语义坍缩，而较大模型（410M、1B）则表现出渐进式退化。频率特异性权衡关系在整个训练过程中保持稳定（r ≈ -0.3）。这些发现表明，语言模型生成文本对语言规律的遵循并非随训练单调增强，而是呈现一个具有最优语义窗口的平衡轨迹。本研究建立了一种评估神经语言模型中涌现语言结构的新方法。
原文摘要 · Abstract (English)
We present the first systematic investigation of Martin's Law - the empirical relationship between word frequency and polysemy - in text generated by neural language models during training. Using DBSCAN clustering of contextualized embeddings as an operationalization of word senses, we analyze four Pythia models (70M-1B parameters) across 30 training checkpoints. Our results reveal a non-monotonic developmental trajectory: Martin's Law emerges around checkpoint 100, reaches peak correlation (r > 0.6) at checkpoint 104, then degrades by checkpoint 105. Smaller models (70M, 160M) experience catastrophic semantic collapse at late checkpoints, while larger models (410M, 1B) show graceful degradation. The frequency-specificity trade-off remains stable (r $\approx$ -0.3) across all models. These findings suggest that compliance with linguistic regularities in LLM-generated text is not monotonically increasing with training, but instead follows a balanced trajectory with an optimal semantic window. This work establishes a novel methodology for evaluating emergent linguistic structure in neural language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。