用物理相变理论分析大模型，发现参数量与生成温度导致两类关键转变。
Phase Transitions in Large Language Models and the $O(N)$ Model
- 将Transformer重构为O(N)模型，用物理相变视角分析大模型行为
- 发现两种相变：参数量突破临界点后能力跃迁，温度变化影响内部表征维度
- 适用于研究模型容量极限和生成质量优化，适合对大模型机制感兴趣的读者
大型语言模型展现出前所未有的丰富缩放行为。在物理学中，缩放行为与相变、临界现象及场论密切相关。为研究大模型中的相变现象，我们将Transformer架构重新表述为O(N)模型。研究揭示了两种不同的相变：分别对应文本生成时使用的温度和模型参数规模。第一种相变可用于估计模型的内部维度，第二种相变具有更高深度，标志着新能力的出现。作为应用，O(N)模型的能量可用来评估大模型参数是否足以学习训练数据。
原文摘要 · Abstract (English)
Large language models (LLMs) exhibit unprecedentedly rich scaling behaviors. In physics, scaling behavior is closely related to phase transitions, critical phenomena, and field theory. To investigate the phase transition phenomena in LLMs, we reformulated the Transformer architecture as an $O(N)$ model. Our study reveals two distinct phase transitions corresponding to the temperature used in text generation and the model's parameter size, respectively. The first phase transition enables us to estimate the internal dimension of the model, while the second phase transition is of \textit{higher-depth} and signals the emergence of new capabilities. As an application, the energy of the $O(N)$ model can be used to evaluate whether an LLM's parameters are sufficient to learn the training data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。