不同模型和数据下，语言模型行为变化规律一致。
Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale
- 用三个简单规则解释98%的词级行为差异。
- 训练中模型逐步更依赖高阶n-gram概率预测。
- 适合关注模型训练本质与通用规律的研究者。
我们发现，无论架构（Transformer、Mamba、RWKV）、训练数据（OpenWebText、The Pile）或规模（1400万至120亿参数），自回归语言模型在预训练过程中表现出高度一致的行为变化模式。基于对超过1400个模型检查点及超过11万词的英文文本分析，我们发现高达98%的词级行为方差可由三个简单启发式规则解释：目标词的独频概率（频率）、该词的n-gram概率，以及该词与其上下文的语义相似度。此外，所有模型均呈现一致的行为阶段，其对词的预测概率随训练进程逐渐更贴合更高阶n-gram概率。这些结果表明，神经语言模型的学习轨迹可能独立于具体模型细节而具有共性。
原文摘要 · Abstract (English)
We show that across architecture (Transformer vs. Mamba vs. RWKV), training dataset (OpenWebText vs. The Pile), and scale (14 million parameters to 12 billion parameters), autoregressive language models exhibit highly consistent patterns of change in their behavior over the course of pretraining. Based on our analysis of over 1,400 language model checkpoints on over 110,000 tokens of English, we find that up to 98% of the variance in language model behavior at the word level can be explained by three simple heuristics: the unigram probability (frequency) of a given word, the $n$-gram probability of the word, and the semantic similarity between the word and its context. Furthermore, we see consistent behavioral phases in all language models, with their predicted probabilities for words overfitting to those words' $n$-gram probabilities for increasing $n$ over the course of training. Taken together, these results suggest that learning in neural language models may follow a similar trajectory irrespective of model details.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。