用预测+动态调度优化GPU集群深度学习任务执行效率
Prediction-Assisted Online Distributed Deep Learning Workload Scheduling in GPU Clusters
- 基于随机森林预测训练迭代次数，动态调整任务分配
- 将复杂调度转为单机最优策略，降低跨服务器通信开销
- 适合频繁重复的分布式深度学习任务场景
深度学习模型的爆发式增长，使得在配备混合并行机制的GPU集群中高效调度分布式深度学习训练任务（DDLwMP）成为迫切需求。本文提出一种自适应最短剩余处理时间优先（A-SRPT）调度算法，这是一种新型的预测辅助在线调度方法，旨在缓解深度学习集群调度的挑战。通过将每个任务建模为对应于异构深度神经网络（DNN）模型及其分布式训练配置的图结构，A-SRPT智能地将任务分配至可用的GPU，从而最小化跨服务器通信开销。鉴于大多数DDLwMP任务具有重复性，A-SRPT引入随机森林回归模型进行训练迭代次数预测。关键在于，A-SRPT将复杂的调度问题转化为单机实例，并通过抢占式‘最短剩余处理时间优先’策略实现最优求解。该优化解作为实际集群任务调度的指导，实现了理论上可证明的竞争性调度效率。我们通过大规模真实测试平台和仿真实验验证了所提算法的有效性。
原文摘要 · Abstract (English)
The recent explosive growth of deep learning (DL) models has necessitated a compelling need for efficient job scheduling for distributed deep learning training with mixed parallelisms (DDLwMP) in GPU clusters. This paper proposes an adaptive shortest-remaining-processing-time-first (A-SRPT) scheduling algorithm, a novel prediction-assisted online scheduling approach designed to mitigate the challenges associated with DL cluster scheduling. By modeling each job as a graph corresponding to heterogeneous Deep Neural Network (DNN) models and their associated distributed training configurations, A-SRPT strategically assigns jobs to the available GPUs, thereby minimizing inter-server communication overhead. Observing that most DDLwMP jobs recur, A-SRPT incorporates a random forest regression model to predict training iterations. Crucially, A-SRPT maps the complex scheduling problem into a single-machine instance, which is addressed optimally by a preemptive "shortest-remaining-processing-time-first" strategy. This optimized solution serves as a guide for actual job scheduling within the GPU clusters, leading to a theoretically provable competitive scheduling efficiency. We conduct extensive real-world testbed and simulation experiments to verify our proposed algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。