发现大模型下游性能可由训练预算直接建模，比旧方法更准。
Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training
- 用简单幂律直接预测模型在多个任务上的准确率随训练规模的变化。
- 在170亿参数、3500亿词数据上验证，比两阶段法外推更可靠。
- 支持复现，适合研究模型缩放规律的学者使用。
尽管传统大模型缩放定律关注预训练损失等代理指标，预测下游任务表现被认为不可靠。本文提出一种直接建模下游基准性能与训练预算关系的框架，发现当词元与参数比例固定时，日志准确率遵循简单的幂律关系，可在多个主流下游任务中准确描述其缩放行为。结果表明，该直接方法的外推性能优于以往的两阶段流程，后者易累积误差。此外，我们引入能预测不同词元-参数比下准确率的函数形式，并考虑重复采样下的推理计算开销。研究在最多170亿参数、3500亿词训练数据、两种数据混合物的模型上进行了验证。为支持可复现性并鼓励未来研究，我们公开了完整的预训练损失与下游评估结果。
原文摘要 · Abstract (English)
While scaling laws for Large Language Models (LLMs) traditionally focus on proxy metrics like pretraining loss, predicting downstream task performance has been considered unreliable. This paper challenges that view by proposing a direct framework to model the scaling of benchmark performance from the training budget. We find that for a fixed token-to-parameter ratio, a simple power law can accurately describe the scaling behavior of log accuracy on multiple popular downstream tasks. Our results show that the direct approach extrapolates better than the previously proposed two-stage procedure, which is prone to compounding errors. Furthermore, we introduce functional forms that predict accuracy across token-to-parameter ratios and account for inference compute under repeated sampling. We validate our findings on models with up to 17B parameters trained on up to 350B tokens across two dataset mixtures. To support reproducibility and encourage future research, we release the complete set of pretraining losses and downstream evaluation results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。