提出可预测混合精度训练时间的模型，误差低于10%。
Training Time Prediction for Mixed Precision-based Distributed Training
- 基于精度感知机制建模分布式训练耗时。
- 在混合精度下误差仅9.8% MAPE，显著优于传统方法。
- 适合资源调度与成本估算场景使用。
分布式深度学习中的训练时间准确预测对资源分配、成本估算和作业调度至关重要。我们发现浮点精度设置是影响训练时间的关键因素，其变化幅度可达最小值的2.4倍。然而，现有分布式训练时间预测研究依赖静态模型计算图，无法捕捉包括混合精度在内的精度变化。根据实验，忽略精度的预测会导致显著误差，平均绝对百分比误差（MAPE）最高达147.85%。为此，我们提出一种精度感知的分布式训练时间预测器，在多种精度设置（包括混合精度）下均实现稳健高精度，平均绝对百分比误差仅为9.8%。
原文摘要 · Abstract (English)
Accurate prediction of training time in distributed deep learning is crucial for resource allocation, cost estimation, and job scheduling. We observe that the floating-point precision setting is a key determinant of training time, leading to training time variations of ~2.4x over its minimum. However, existing studies on distributed training time prediction rely on static model computation graphs that do not capture precision variations, including mixed precision. According to our experiments, training time prediction without considering precision results in significant prediction errors - reaching up to 147.85% in mean absolute percentage error (MAPE). To address this issue, we propose a precision-aware distributed training time predictor that achieves robust accuracy across diverse precision settings, including mixed precision, with 9.8% MAPE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。