自动规划大模型并行策略，提升训练效率
Automatically Planning Optimal Parallel Strategy for Large Language Models
- 通过解耦计算、通信与重叠时间，建立训练时长模拟模型
- 实测预测准确率达96%，推荐策略始终全局最优
- 适合大规模模型训练团队快速部署高效并行方案
基于Transformer的大规模语言模型参数量持续增长，计算集群规模也随之扩大。快速调动大量计算资源进行并行计算的技术日益重要。本文提出一种自动并行算法，可根据模型与硬件信息自动规划最大吞吐量的并行策略。通过将训练时长解耦为计算、通信和重叠三部分，构建训练时长仿真模型，并基于该模型剪枝并行解空间以缩短搜索时间。多节点实验结果表明，该算法可实时估算并行训练时长，平均准确率达96%。在测试中，算法推荐的策略始终为全局最优。
原文摘要 · Abstract (English)
The number of parameters in large-scale language models based on transformers is gradually increasing, and the scale of computing clusters is also growing. The technology of quickly mobilizing large amounts of computing resources for parallel computing is becoming increasingly important. In this paper, we propose an automatic parallel algorithm that automatically plans the parallel strategy with maximum throughput based on model and hardware information. By decoupling the training time into computation, communication, and overlap, we established a training duration simulation model. Based on this simulation model, we prune the parallel solution space to shorten the search time required. The multi-node experiment results show that the algorithm can estimate the parallel training duration in real time with an average accuracy of 96%. In our test, the recommendation strategy provided by the algorithm is always globally optimal.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。