构建4000+模型数据集，揭示训练设计对缩放定律的影响。
Gemstones: A Model Suite for Multi-Faceted Scaling Laws
- 基于多种架构和超参数组合训练模型，研究缩放规律。
- 发现缩放定律结果高度依赖实验设计与具体模型选择。
- 开源包含20亿参数模型的Gemstones数据集，支持复杂分析。
缩放定律通常在一组固定超参数的模型中拟合。本文通过多类架构和多样超参数设置研究缩放规律，揭示其对最终结论的影响。作为主要成果，我们发布Gemstones：一个包含超过4000个检查点的开源缩放定律数据集，涵盖最多20亿参数的Transformer模型，以及学习率与退火策略的消融实验。这些检查点支持更复杂的缩放研究，如宽度与深度的关系分析。通过分析该模型套件，我们发现缩放定律的建议结果对实验设计过程及用于拟合的具体模型检查点极为敏感。
原文摘要 · Abstract (English)
Scaling laws are typically fit using a family of models with a narrow range of frozen hyperparameter choices. In this work we study scaling laws using multiple architectural shapes and hyperparameter choices, highlighting their impact on resulting prescriptions. As a primary artifact of our research, we release the Gemstones: an open-source scaling law dataset, consisting of over 4000 checkpoints from transformers with up to 2 billion parameters and diverse architectural shapes; including ablations over learning rate and cooldown. Our checkpoints enable more complex studies of scaling, such as analyzing the relationship between width and depth. By examining our model suite, we find that the prescriptions of scaling laws can be highly sensitive to the experimental design process and the specific model checkpoints used during fitting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。