用10%预算达成接近全量实验的精度，智能选实验省下百万训练成本
Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection
- 根据实验成本动态选择最有价值的训练任务，提升预算利用率
- 在多个任务上仅用10%预算即逼近全量实验的外推精度
- 适合需要大规模训练但预算有限的研究团队快速验证模型规律
Scaling laws 被用于规划数百万美元的训练任务，但拟合这些规律本身也可能耗资数百万。在现代大规模训练流程中，获取足够信息的先导实验集已不再是常规预处理步骤，而成为重大的预算分配难题。本文将 scaling-law 拟合问题建模为预算感知的顺序实验设计：给定一组异构成本的可运行实验，如何选择执行哪些实验以最大化高成本目标区域的外推准确性。我们提出 $<math xmlns='http://www.w3.org/1998/Math/MathML' style='font-size:1.2em;'\u003eSL^2</math>（Scaling Laws, Spend Less），一种基于不确定性的方法，用于在序列中分配实验预算，优先选择对目标区域外推最有帮助的实验。在涵盖多种 scaling-law 任务的多样化基准测试中，$<math xmlns='http://www.w3.org/1998/Math/MathML' style='font-size:1.2em;'\u003eSL^2</math> 显著优于经典设计基线，并常达到使用完整实验集拟合的性能水平，同时仅消耗约10%的总训练预算。代码已开源：https://github.com/PlanarG/active-sl。
原文摘要 · Abstract (English)
Scaling laws are used to plan multi-million-dollar training runs, but fitting those laws can itself cost millions. In modern large-scale workflows, assembling a sufficiently informative set of pilot experiments is already a major budget-allocation problem rather than a routine preprocessing step. We formulate scaling-law fitting as budget-aware sequential experimental design: given a finite pool of runnable experiments with heterogeneous costs, choose which runs to execute so as to maximize extrapolation accuracy in a high-cost target region. We propose $\mathrm{SL}^2$ (Scaling Laws, Spend Less), an uncertainty-aware method for sequentially allocating experimental budget toward the runs most useful for target-region extrapolation. Across a diverse benchmark of scaling-law tasks, $\mathrm{SL}^2$ outperforms classical design-based baselines, and often approaches the performance of fitting on the full experimental set while using only about 10\% of the total training budget. Our code is available at https://github.com/PlanarG/active-sl.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。