揭秘大模型预训练的高效扩展方法，提升GPU利用率
Scaling Performance of Large Language Model Pretraining
- 提出分布式训练优化方案，高效管理跨数百节点的大数据集
- 通过强化数据并行性，实现对可用GPU算力的充分释放
- 为大规模模型训练提供可复现的实践指南，适合工程团队参考
大型语言模型（LLMs）在众多自然语言处理任务中表现出色。然而，训练这些模型计算成本极高，前沿人工智能公司正投入数十亿美元建设超级计算机基础设施，以在日益庞大的数据集上训练更大规模的模型。遗憾的是，关于这些大规模训练流程的扩展性能与训练考量的公开信息极少。使用超大规模数据集和模型具有高度复杂性，现有公开文献中缺乏针对扩展大语言模型时提升训练性能的实际建议。本文旨在部分揭示大语言模型预训练流程——特别是分布式训练、跨数百节点管理大规模数据集，以及在强调充分利用可用GPU算力的前提下，扩展数据并行性的方法。
原文摘要 · Abstract (English)
Large language models (LLMs) show best-in-class performance across a wide range of natural language processing applications. Training these models is an extremely computationally expensive task; frontier Artificial Intelligence (AI) research companies are investing billions of dollars into supercomputing infrastructure to train progressively larger models on increasingly massive datasets. Unfortunately, very little information about the scaling performance and training considerations of these large training pipelines is released publicly. Working with very large datasets and models can be complex and practical recommendations are scarce in the public literature for tuning training performance when scaling up large language models. In this paper, we aim to demystify the large language model pretraining pipeline somewhat - in particular with respect to distributed training, managing large datasets across hundreds of nodes, and scaling up data parallelism with an emphasis on fully leveraging available GPU compute capacity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。