用小模型阶梯预测大模型任务表现，省99%算力
Establishing Task Scaling Laws via Compute-Efficient Model Ladders
- 构建模型阶梯，分两步预测任务性能：先估中间损失，再推准确率
- 对7B和13B模型预测误差仅2个百分点，验证了方法有效性
- 适合需要高效评估大模型性能的研究者，尤其关注算力受限场景
我们提出任务缩放定律与模型阶梯方法，用于预测预训练语言模型在过训练状态下的单个任务表现。标准语言建模损失的幂律无法准确刻画任务性能，因此采用两步预测法：(1) 利用模型和数据规模预测中间损失；(2) 基于此预测任务表现。我们训练了一组小规模的“阶梯”模型，收集数据点以拟合两个预测步骤的参数化函数，并对两个目标模型进行预测：一个训练至4万亿令牌的70亿参数模型，以及一个训练至5万亿令牌的130亿参数模型。训练阶梯模型仅消耗目标模型所需算力的1%。在四个以排序分类格式呈现的多项选择任务上，我们对两个目标模型的预测准确率误差均在2个百分点以内。我们发现，预测误差较高的任务其指标在模型检查点间方差也更大。此外，我们对比了多种准确率预测设计选择，并提出了扩展该方法至新模型与任务的建议。
原文摘要 · Abstract (English)
We develop task scaling laws and model ladders to predict the individual task performance of pretrained language models (LMs) in the overtrained setting. Standard power laws for language modeling loss cannot accurately model task performance. Therefore, we leverage a two-step prediction approach: (1) use model and data size to predict an intermediate loss, then (2) use it to predict task performance. We train a set of small-scale "ladder" models, collect data points to fit the parameterized functions of the two prediction steps, and make predictions for two target models: a 7B model trained to 4T tokens and a 13B model trained to 5T tokens. Training the ladder models only costs 1% of the compute used for the target models. On four multiple-choice tasks formatted as ranked classification, we can predict the accuracy of both target models within 2 points of absolute error. We find that tasks with higher prediction error also have higher variance in the metrics over model checkpoints. We also contrast multiple design choices for predicting accuracy, and present recommendations for extending our method to new models and tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。