发现大模型训练的最优批量大小主要随数据量增长，而非模型大小。
How Does Critical Batch Size Scale in Pre-training?
- 通过系统实验测量不同规模模型的临界批量大小
- 发现临界批量大小与数据量呈正相关，与模型大小关系不大
- 强调超参设计对大规模预训练的重要性，适合模型训练优化研究者
在给定资源下训练大规模模型需精心设计并行策略。关键批量大小（CBS）衡量时间与算力间的权衡，标志着数据并行带来的收益开始递减的阈值。本文提出一种CBS度量方法，并在C4数据集上预训练了一系列自回归语言模型（参数量从8500万到12亿不等）。通过大量超参调优及对批量大小、动量、学习率及其调度的精细控制，系统研究了模型规模对CBS的影响。随后，基于模型和数据规模拟合缩放定律，分离二者效应。结果表明，CBS主要随数据规模增长，而非模型规模。这一发现通过神经网络无限宽度极限与无穷维最小二乘回归的理论分析得以验证。此外，本文还强调了常见超参选择在大规模预训练研究中的关键作用，尤其在非固定训练时长场景下。
原文摘要 · Abstract (English)
Training large-scale models under given resources requires careful design of parallelism strategies. In particular, the efficiency notion of critical batch size (CBS), concerning the compromise between time and compute, marks the threshold beyond which greater data parallelism leads to diminishing returns. To operationalize it, we propose a measure of CBS and pre-train a series of auto-regressive language models, ranging from 85 million to 1.2 billion parameters, on the C4 dataset. Through extensive hyper-parameter sweeps and careful control of factors such as batch size, momentum, and learning rate along with its scheduling, we systematically investigate the impact of scale on CBS. Then we fit scaling laws with respect to model and data sizes to decouple their effects. Overall, our results demonstrate that CBS scales primarily with data size rather than model size, a finding we justify theoretically through the analysis of infinite-width limits of neural networks and infinite-dimensional least squares regression. Of independent interest, we highlight the importance of common hyper-parameter choices and strategies for studying large-scale pre-training beyond fixed training durations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。