考虑推理成本后,过度训练反而是最优选择。
Test-Time Scaling Makes Overtraining Compute-Optimal
- 提出联合优化预训练与测试时扩增的新型缩放规律
- 在8个任务中发现最优预训练已进入过度训练区
- 适合关注推理效率与模型性能平衡的研究者
现代大模型常通过重复采样等方式进行测试时扩展,导致推理成本随模型规模和采样次数增加。这引入了预训练与推理之间的权衡,而现有预训练缩放规律(如Chinchilla)并未涵盖此问题。本文提出训练-测试($T^2$)缩放规律,联合优化模型大小、训练数据量与推理采样数,在固定端到端预算下实现最优配置。$T^2$将测试时缩放中的pass@$k$建模融入预训练决策,预测结果在多种建模方法下均稳健:既评估任务损失的联合缩放效应,也建模准确率影响。在8个下游任务中,考虑推理成本后,最优预训练策略显著转向过度训练区域,远超传统缩放方案范围。通过在$T^2$预测的最优区域预训练高度过训练模型,验证其性能显著优于仅依赖预训练缩放的方案。此外,即使在前沿模型经历后训练后,该结论依然成立,表明$T^2$对现代部署具有实际意义。
原文摘要 · Abstract (English)
Modern LLMs scale at test-time, e.g. via repeated sampling, where inference cost grows with model size and the number of samples. This creates a trade-off that pretraining scaling laws, such as Chinchilla, do not address. We present Train-to-Test ($T^2$) scaling laws that jointly optimize model size, training tokens, and number of inference samples under fixed end-to-end budgets. $T^2$ modernizes pretraining scaling laws with pass@$k$ modeling used for test-time scaling, then jointly optimizes pretraining and test-time decisions. Forecasts from $T^2$ are robust over distinct modeling approaches: measuring joint scaling effect on the task loss and modeling impact on task accuracy. Across eight downstream tasks, we find that when accounting for inference cost, optimal pretraining decisions shift radically into the overtraining regime, well-outside of the range of standard pretraining scaling suites. We validate our results by pretraining heavily overtrained models in the optimal region that $T^2$ scaling forecasts, confirming their substantially stronger performance compared to pretraining scaling alone. Finally, as frontier LLMs are post-trained, we show that our findings survive the post-training stage, making $T^2$ scaling meaningful in modern deployments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。