通过波前调度提升多任务多模态大模型分布式训练效率
Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Scheduling
- 将模型执行分解为波次，分步优化异构负载与依赖关系
- 相比现有系统最高提速71%，显著提升训练效率
- 适合需要高效训练多任务多模态模型的研究者
近期的基础模型能够通过统一的主干结构和多个专用组件处理多种任务与数据模态。然而，由于模型架构复杂且不同任务与模态的工作负载异质性高，此类多任务（MT）多模态（MM）模型的高效训练面临重大系统挑战。本文提出Spindle，一种专为资源高效、高性能训练MT MM模型设计的新训练系统，采用波前调度策略。其核心思想是将模型执行分解为多个波次，顺序求解联合优化问题，包括感知异构性的负载并行化和基于依赖关系的执行调度。我们在多种MT MM模型上构建并评估该系统。实验表明，Spindle性能优越，相比最先进训练系统最高提速71%。
原文摘要 · Abstract (English)
Recent foundation models are capable of handling multiple tasks and multiple data modalities with the unified base model structure and several specialized model components. However, efficient training of such multi-task (MT) multi-modal (MM) models poses significant system challenges due to the sophisticated model architecture and the heterogeneous workloads of different tasks and modalities. In this paper, we propose Spindle, a brand new training system tailored for resource-efficient and high-performance training of MT MM models via wavefront scheduling. The key idea of Spindle is to decompose the model execution into waves and address the joint optimization problem sequentially, including both heterogeneity-aware workload parallelization and dependency-driven execution scheduling. We build our system and evaluate it on various MT MM models. Experiments demonstrate the superior performance and efficiency of Spindle, with speedup ratio up to 71% compared to state-of-the-art training systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。