通过45组训练实验，揭示大模型预训练的稳定性与异常情况
PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs
- 在5种规模模型上运行9个随机种子，共生成7000个检查点
- 发现不同初始条件下的训练动态高度一致，存在可识别的异常训练路径
- 适合关注模型训练可靠性与鲁棒性的研究者参考
语言模型预训练的稳定性及其对下游性能的影响仍缺乏深入研究。已有工作表明，微小的初始条件变化（如随机种子）可能导致显著不同的训练结果。然而，研究社区仍缺乏系统性工具和资源来深入分析解码器单向语言模型的预训练稳定性。本文提出PolyPythias，包含45组新的Pythia模型预训练实验：在5种模型规模（14M至410M参数）下，每种使用9个随机种子，共产生约7000个新检查点并公开发布。结合原有的5组实验，我们研究了由种子决定的初始条件（包括参数初始化和数据顺序）对下游性能、学习到的语言表征以及训练阶段涌现行为的影响。分析显示，除常见的缩放规律外，不同模型规模和初始条件下训练动态高度一致。此外，每种模型的新种子使我们能识别出异常训练路径，并刻画其特征。结果表明，这些方法具有预测训练稳定性的潜力。
原文摘要 · Abstract (English)
The stability of language model pre-training and its effects on downstream performance are still understudied. Prior work shows that the training process can yield significantly different results in response to slight variations in initial conditions, e.g., the random seed. Crucially, the research community still lacks sufficient resources and tools to systematically investigate pre-training stability, particularly for decoder-only language models. We introduce the PolyPythias, a set of 45 new training runs for the Pythia model suite: 9 new seeds across 5 model sizes, from 14M to 410M parameters, resulting in about 7k new checkpoints that we release. Using these new 45 training runs, in addition to the 5 already available, we study the effects of different initial conditions determined by the seed -- i.e., parameters' initialisation and data order -- on (i) downstream performance, (ii) learned linguistic representations, and (iii) emergence of training phases. In addition to common scaling behaviours, our analyses generally reveal highly consistent training dynamics across both model sizes and initial conditions. Further, the new seeds for each model allow us to identify outlier training runs and delineate their characteristics. Our findings show the potential of using these methods to predict training stability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。