用空档期执行其他任务，让大模型训练的显卡更高效
PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Training
- 利用训练间隙的空闲时间运行其他任务
- 大模型训练时显卡利用率最高提升63%，主任务仅慢2%
- 适合大规模训练团队降低算力浪费
训练包含数十亿参数的深度神经网络通常采用流水线并行（PP）方式。然而，由于流水线空档（bubble）导致的显卡闲置，使大规模训练的显卡利用率低下，空档时间常达15%-30%，甚至超过60%。本文提出PipeFill，通过在流水线空档期执行其他待处理任务来填充空闲显卡。为实现低开销切换与高效率填充，PipeFill根据实测空档时长和显存可用性精确匹配填充任务，引入显式流水线空档指令，并协调填充任务的调度与执行。实验表明，对于大规模大模型训练，PipeFill可将整体显卡利用率提升高达63%，主训练任务延迟低于2%；在8000张显卡上，相当于多完成2600张显卡的工作量。小规模训练中仍可实现5%-15%的利用率提升。
原文摘要 · Abstract (English)
Training Deep Neural Networks (DNNs) with billions of parameters generally involves pipeline-parallel (PP) execution. Unfortunately, PP model training can use GPUs inefficiently, especially at large scale, due to idle GPU time caused by pipeline bubbles, which are often 15-30% and can exceed 60% of the training job's GPU allocation. To improve the GPU utilization of PP model training, this paper describes PipeFill, which fills pipeline bubbles with execution of other pending jobs. By leveraging bubble GPU time, PipeFill reduces the GPU utilization sacrifice associated with scaling-up of large-model training. To context-switch between fill jobs and the main training job with minimal overhead to the main job, and maximize fill job efficiency, PipeFill carefully fits fill job work to measured bubble durations and GPU memory availability, introduces explicit pipeline-bubble instructions, and orchestrates placement and execution of fill jobs in pipeline bubbles. Experiments show that PipeFill can increase overall utilization by up to 63% for GPUs used in large-scale LLM training, with <2% slowdown of the training job, and 5-15% even for low-scale LLM training. For large-scale LLM training on 8K GPUs, the 63% increase translates to up to 2.6K additional GPUs worth of work completed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。