arXiv:2412.01212stat.MLcond-mat.stat-mech2024-12中稿 · publication in PRE被引 1

发现语言模型中存在布里渊-科斯特利茨-索利斯相变,揭示语言结构的普适物理规律。

Berezinskii--Kosterlitz--Thouless transition in a context-sensitive random language model

  • 构建上下文敏感语言模型,通过符号频率偏移作为序参量探测相变
  • 在句子无限长极限下,序参量从零突变为非零,证明数学奇点存在
  • 识别为布里渊-科斯特利茨-索利斯型相变,适合研究语言与物理系统关联者

自然语言中长期观察到多种幂律临界特性,类似物理系统在相变点附近的标度行为。大语言模型的兴起进一步激发了这种类比,展现出缩放律和涌现能力等物理概念的相似性。然而,统计物理中定义的相变在生成式语言模型中的具体实例仍缺乏。本文受一维庞茨模型启发,构建了一种属于上下文敏感语法类的简单概率语言模型,称为上下文敏感随机语言模型,并在数值上明确验证了自然语言模型框架下的相变。我们明确展示了定义良好的序参量——捕获生成句子中符号频率偏移——在句子长度趋于无穷时,从严格零跃变为严格非零值,表明所考虑的随机语言模型参数调节下出现数学奇点。此外,我们确认该相变为布里渊-科斯特利茨-索利斯(BKT)相变的一种变体,其临界性质不仅出现在转变点,还贯穿整个相区。这一发现表明,自然语言中的临界特性可能无需精细调优或自组织临界性,而是由语言结构与BKT相之间的内在联系普遍解释。

原文摘要 · Abstract (English)

Several power-law critical properties involving different statistics in natural languages -- reminiscent of scaling properties of physical systems at or near phase transitions -- have been documented for decades. The recent rise of large language models has added further evidence and excitement by providing intriguing similarities with notions in physics such as scaling laws and emergent abilities. However, specific instances of classes of generative language models that exhibit phase transitions, as understood by the statistical physics community, are lacking. In this work, inspired by the one-dimensional Potts model in statistical physics, we construct a simple probabilistic language model that falls under the class of context-sensitive grammars, which we call the context-sensitive random language model, and numerically demonstrate an unambiguous phase transition in the framework of a natural language model. We explicitly show that a precisely defined order parameter -- that captures symbol frequency biases in the sentences generated by the language model -- changes from strictly zero to a strictly nonzero value (in the infinite-length limit of sentences), implying a mathematical singularity arising when tuning the parameter of the stochastic language model we consider. Furthermore, we identify the phase transition as a variant of the Berezinskii--Kosterlitz--Thouless (BKT) transition, which is known to exhibit critical properties not only at the transition point but also in the entire phase. This finding leads to the possibility that critical properties in natural languages may not require careful fine-tuning nor self-organized criticality, but are generically explained by the underlying connection between language structures and the BKT phases.

语言模型相变物理类比BKT

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。