Astra自动搜索异构显卡上的高效省钱并行策略。
Astra: Efficient and Money-saving Automatic Parallel Strategies Search on Heterogeneous GPUs
- 基于数学模型自动搜寻最优显卡配置与并行参数
- 单卡搜索仅需1.27秒，异构环境平均<1.35分钟，准确率超95%
- 首个兼顾性能与成本的自动化并行策略搜索框架
本文提出一种高效且节省成本的异构GPU上自动并行策略搜索框架Astra。Astra首先在显卡类型和数量的配置空间以及并行参数空间中搜索效率最优的并行策略；其次通过数学建模精确估算异构训练的时间消耗；最后首次实现面向经济成本的自动并行策略搜索。实验表明，Astra的吞吐量优于人工设计策略。在单卡设置下，搜索耗时仅为1.27秒；在异构GPU设置下，平均搜索时间低于1.35分钟，且精度超过95%。
原文摘要 · Abstract (English)
In this paper, we introduce an efficient and money-saving automatic parallel strategies search framework on heterogeneous GPUs: Astra. First, Astra searches for the efficiency-optimal parallel strategy in both GPU configurations search space (GPU types and GPU numbers) and parallel parameters search space. Then, Astra also provides the solution on heterogeneous GPUs by mathematically modeling the time consumption of heterogeneous training. At last, Astra is the first to propose the automatic parallel strategy search on money-saving. The experiment results demonstrate that Astra can achieve better throughput than expert-designed strategies. The search time cost for Astra can also be limited to 1.27 seconds in a single-GPU setting and less than 1.35 minutes in a heterogeneous-GPU setting on average with an accuracy of over 95%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。