ScaleDL精准预测分布式深度学习运行时,兼顾精度与成本。
ScaleDL: Towards Scalable and Efficient Runtime Prediction for Distributed Deep Learning Workloads
- 用非线性分层建模+图神经网络捕捉跨层交互
- 相比基线模型,误差降低6倍(MRE)和5倍(RMSE)
- 适合需要高效资源调度的AI系统研发人员
深度神经网络(DNN)是现代AI服务的核心,支撑自动驾驶、聊天机器人和推荐系统等应用。随着模型规模与复杂度上升,训练与推理任务对分布式计算资源提出前所未有的需求,准确的运行时预测对优化开发与资源分配至关重要。传统方法依赖加性计算单元模型,限制了精度与泛化能力;而图增强建模虽提升性能,但显著增加数据收集成本。为此,我们提出ScaleDL,一种结合非线性分层建模与图神经网络(GNN)跨层交互机制的运行时预测框架,实现不同网络架构间的层次化泛化。同时,采用D-最优方法降低数据收集成本。在五个主流DNN模型工作负载上的实验表明,ScaleDL显著提升预测精度与泛化能力,相比基线模型,平均相对误差(MRE)降低6倍,均方根误差(RMSE)降低5倍。
原文摘要 · Abstract (English)
Deep neural networks (DNNs) form the cornerstone of modern AI services, supporting a wide range of applications, including autonomous driving, chatbots, and recommendation systems. As models increase in size and complexity, DNN workloads such as training and inference tasks impose unprecedented demands on distributed computing resources, making accurate runtime prediction essential for optimizing development and resource allocation. Traditional methods rely on additive computational unit models, limiting their accuracy and generalizability. In contrast, graph-enhanced modeling improves performance but significantly increases data collection costs. Therefore, there is a critical need for a method that strikes a balance between accuracy, generalizability, and data collection costs. To address these challenges, we propose ScaleDL, a novel runtime prediction framework that combines nonlinear layer-wise modeling with graph neural network (GNN)-based cross-layer interaction mechanism, enabling accurate DNN runtime prediction and hierarchical generalizability across different network architectures. Additionally, we employ the D-optimal method to reduce data collection costs. Experiments on the workloads of five popular DNN models demonstrate that ScaleDL enhances runtime prediction accuracy and generalizability, achieving 6 times lower MRE and 5 times lower RMSE compared to baseline models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。