利用流水线训练中的空档提升GPU利用率，降低大模型训练成本
FreeRide: Harvesting Bubbles in Pipeline Parallelism
- 通过智能调度捕捉流水线中的计算空档执行额外任务
- 在不显著影响训练的前提下实现平均7.8%成本节约
- 适合需要高效利用算力的大模型训练与多任务并行场景
流水线并行中的计算空档（bubbles）是大语言模型训练时间超过40%的主因，导致GPU资源严重浪费。本文提出FreeRide系统，通过自动识别和管理这些不规则分布的空档，将它们用于运行模型训练、图分析和图像处理等侧任务。该系统提供易用接口，动态协调侧任务与主训练对GPU资源的竞争，实现约1%的极低开销。实验表明，FreeRide在保持训练稳定的同时，平均节省7.8%的训练成本，有效提升了硬件资源利用率。
原文摘要 · Abstract (English)
The occurrence of bubbles in pipeline parallelism is an inherent limitation that can account for more than 40% of the large language model (LLM) training time and is one of the main reasons for the underutilization of GPU resources in LLM training. Harvesting these bubbles for GPU side tasks can increase resource utilization and reduce training costs but comes with challenges. First, because bubbles are discontinuous with various shapes, programming side tasks becomes difficult while requiring excessive engineering effort. Second, a side task can compete with pipeline training for GPU resources and incur significant overhead. To address these challenges, we propose FreeRide, a system designed to harvest bubbles in pipeline parallelism for side tasks. FreeRide provides programmers with interfaces to implement side tasks easily, manages bubbles and side tasks during pipeline training, and controls access to GPU resources by side tasks to reduce overhead. We demonstrate that FreeRide achieves 7.8% average cost savings with a negligible overhead of about 1% in training LLMs while serving model training, graph analytics, and image processing side tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。