提出三种生成式评估的预训练扩展规律,可预测大模型表现。
Pretraining Scaling Laws for Generative Evaluations of Language Models
- 用计算量、参数+数据量、参考答案似然三类变量建模生成任务扩展规律。
- 参考答案似然规律在5个数量级上稳定,远超其他两种规律的1.5~2.5个数量级。
- 能用低成本模型预测高成本模型在数学/编程等生成任务上的通过率。
神经网络的缩放定律推动了参数量、数据量和算力的指数增长。尽管预训练损失和判别式基准的缩放行为已明确,但数学解题或软件工程等生成式基准仍缺乏研究。本文提出并评估三种用于拟合生成评估中pass-at-$k$表现的预训练缩放定律,并用低成本模型预测最昂贵模型的pass-at-$k$表现。三种定律分别基于:(1) 预训练算力,(2) 模型参数与预训练数据量,(3) 参考正确解的对数似然。首先,我们发现生成评估引入新超参数$ k $,可调节缩放行为及性能可预测性;其次,参数稳定性差异显著:前两种定律仅在最后约1.5~2.5个数量级稳定,而参考答案似然定律在约5个数量级内保持收敛;第三,在预测性能上,三者表现相近,但算力定律对小$ k $预测略差,参考答案似然定律对大$ k $预测略差。最后,我们建立了理论联系,证明算力定律是参数+数据量定律的算力最优包络。该框架为研究人员提供预测生成性能的方法,加速具备推理、求解与创造能力模型的发展。
原文摘要 · Abstract (English)
Neural scaling laws have driven the field's ever-expanding exponential growth in parameters, data and compute. While scaling behaviors for pretraining losses and discriminative benchmarks are well established, generative benchmarks such as mathematical problem-solving or software engineering remain under-explored. We propose and evaluate three different pretraining scaling laws for fitting pass-at-$k$ on generative evaluations and for predicting pass-at-$k$ of the most expensive model using cheaper models. Our three scaling laws differ in the covariates used: (1) pretraining compute, (2) model parameters and pretraining tokens, (3) log likelihoods of gold reference solutions. First, we demonstrate that generative evaluations introduce new hyperparameters (in our setting, $k$) that act as a control lever for scaling behavior, modulating both the scaling law parameters and the predictability of performance. Second, we identify a stark difference in parameter stability: while the compute and parameters+tokens laws stabilize for only the last $\mathord{\sim}1.5\mathord{-}2.5$ orders of magnitude, the gold reference likelihood law is uniquely stable, converging across $\mathord{\sim}5$ orders. Third, in terms of predictive performance, we find all three scaling laws perform comparably, although the compute law predicts slightly worse for small $k$ and the gold reference law predicts slightly worse for large $k$. Finally, we establish a theoretical connection, proving that the compute scaling law emerges as the compute-optimal envelope of the parameters-and-tokens law. Our framework provides researchers and practitioners with insights and methodologies to forecast generative performance, accelerating progress toward models that can reason, solve, and create.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。