用贝叶斯优化减少训练量，高效构建大模型缩放定律。
Amortizing Scaling Law Construction Costs
- 将数据收集建模为贝叶斯优化问题，智能选择配置点。
- 在有限算力下，计算节省10到100倍，逼近全网格精度。
- 适合资源受限下设计大模型的算法工程师和研究者。
缩放定律指导大模型训练设计，但传统方法需在超参数、训练样本数和参数量上遍历所有组合，计算开销巨大。实际上，拟合缩放定律仅需各算力水平下的最优损失前沿，其余配置可丢弃。本文提出一种高效缩放定律构建框架，将数据收集建模为贝叶斯优化问题，并定义新指标比较不同拟合方法在算力受限下的表现。发现按算力顺序逐步扩展搜索范围，可显著提升恢复效率。通过引入代理幻想评估补充观测配置，能重建完整实验网格，实现无需训练全部配置即可准确拟合缩放定律。该方法在计算成本降低10至100倍的同时，仍能接近全密集网格的拟合效果。
原文摘要 · Abstract (English)
Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and parameter counts, which is computationally expensive. Fitting a scaling law, however, only requires the best-loss frontier across compute scales, discarding most of the trained configurations. We propose a framework for efficient scaling law construction that formulates data collection as a Bayesian optimization problem, and introduce metrics for comparing scaling law fitting methods under constrained compute budgets. We find that progressively expanding the compute budget during acquisition, mirroring the compute-ordered evaluation of configurations in practice, substantially improves recovery efficiency. Augmenting the observed configurations with surrogate-fantasized evaluations then recovers the broader experimental grid, allowing accurate scaling law fitting without training every configuration. Together, these can closely match scaling law fits over a full dense grid at computational savings of up to $10\text{--}100\times$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。