用简单幂律关系预测不同数据集的训练损失，扩展了缩放定律的应用范围。
Loss-to-Loss Prediction: Scaling Laws for All Datasets
- 基于训练算力对齐模型，建立跨数据集训练损失的幂律关联。
- 在20倍最大算力预算下仍能准确外推，关键关系具有普适性。
- 适用于代码与非代码数据差异大的预训练数据，适合做模型性能预估。
虽然缩放定律能可靠预测单一数据分布下不同计算规模的训练损失，但关于分布变化时预测应如何调整仍不明确。本文提出一种从一个损失预测另一个损失的策略，应用于跨预训练数据集及从预训练数据到下游任务数据的损失预测。预测结果在最大使用算力20倍的情况下仍具良好外推性。我们发现存在简单的偏移幂律关系：（1）相同训练算力下两模型在不同数据集上的训练损失（训练到训练）；（2）单模型在任意下游分布上的训练损失与测试损失（训练到测试）；（3）两模型在不同训练数据集上对应的测试损失（测试到测试）。这些关系在差异显著的预训练数据（如全代码与无代码数据）及多种下游任务中均成立。某些场景下，该方法比单一数据集缩放定律预测更准确。
原文摘要 · Abstract (English)
While scaling laws provide a reliable methodology for predicting train loss across compute scales for a single data distribution, less is known about how these predictions should change as we change the distribution. In this paper, we derive a strategy for predicting one loss from another and apply it to predict across different pre-training datasets and from pre-training data to downstream task data. Our predictions extrapolate well even at 20x the largest FLOP budget used to fit the curves. More precisely, we find that there are simple shifted power law relationships between (1) the train losses of two models trained on two separate datasets when the models are paired by training compute (train-to-train), (2) the train loss and the test loss on any downstream distribution for a single model (train-to-test), and (3) the test losses of two models trained on two separate train datasets (test-to-test). The results hold up for pre-training datasets that differ substantially (some are entirely code and others have no code at all) and across a variety of downstream tasks. Finally, we find that in some settings these shifted power law relationships can yield more accurate predictions than extrapolating single-dataset scaling laws.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。