研究大模型训练中批量大小对效果的影响，给出资源约束下的优化方案。
Scaling Law for Language Models Training Considering Batch Size
- 通过实验建立模型规模与数据量的基准缩放定律。
- 发现批量大小在固定算力或固定数据量下各有不同缩放规律。
- 结果可指导实际训练中资源分配，适合模型训练工程师参考。
近年来，大语言模型（LLMs）取得了显著进展，缩放定律在其中起到了关键作用。本文通过实证研究，探讨了一个关键超参数——全局批量大小对LLM训练过程的影响。我们训练了从1.25亿到26亿参数的语言模型，使用高达3000亿高质量标记样本。实验建立了模型规模与训练数据量的基本缩放定律，并进一步分析了批量大小和学习率变化对模型收敛性和泛化能力的影响。研究得出了两种情形下的批量大小缩放定律：在固定计算预算下和固定训练数据量下。通过对更大规模模型的外推实验验证了预测规律，为特定资源约束下的大模型训练策略优化提供了依据。
原文摘要 · Abstract (English)
Large language models (LLMs) have made remarkable advances in recent years, with scaling laws playing a critical role in this rapid progress. In this paper, we empirically investigate how a critical hyper-parameter, i.e., the global batch size, influences the LLM training prdocess. We begin by training language models ranging from 125 million to 2.6 billion parameters, using up to 300 billion high-quality tokens. Through these experiments, we establish a basic scaling law on model size and training data amount. We then examine how varying batch sizes and learning rates affect the convergence and generalization of these models. Our analysis yields batch size scaling laws under two different cases: with a fixed compute budget, and with a fixed amount of training data. Extrapolation experiments on models of increasing sizes validate our predicted laws, which provides guidance for optimizing LLM training strategies under specific resource constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。