解决大模型训练中的数据不均衡问题，提升训练效率1.32至2.66倍。
Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment
- 动态调整并行策略以适应序列长度变化
- 两阶段数据分配使模型副本间负载更均衡
- 适合大规模Transformer训练系统优化
为优化大Transformer模型的训练，高效并行计算与先进数据管理不可或缺。然而，现有方法常假设训练工作量稳定均匀，忽视了采样和打包过程引发的数据不均衡问题。其中，数据采样不均衡源于训练数据序列长度分布不均；数据打包不均衡则来自注意力机制的线性内存复杂度与二次时间复杂度之间的差异。为此，我们提出Hydraulis，联合优化并行策略与数据分配。一方面，针对训练中跨迭代的序列长度变化，采用动态异构并行策略；另一方面，设计两阶段数据分配方案，在模型副本内部与之间实现训练负载的均衡。实证结果表明，Hydraulis相比现有系统性能提升1.32至2.66倍。
原文摘要 · Abstract (English)
To optimize large Transformer model training, both efficient parallel computing and advanced data management are indispensable. However, current methods often assume a stable and uniform training workload, neglecting data-induced imbalances-arising from both sampling and packing processes-which can impede training performance. Specifically, data sampling imbalance arises from uneven sequence length distribution of the training data, while data packing imbalance stems from the discrepancy between the linear memory complexity and quadratic time complexity of the attention mechanism. To address these imbalance issues, we develop Hydraulis, which jointly optimizes the parallel strategies and data assignment. For one thing, we introduce large model training with dynamic heterogeneous parallel strategies in response to the sequence length variations within and across training iterations. For another, we devise a two-stage data assignment approach, which strikes a good balance in terms of the training workloads both within and across model replicas. Empirical results demonstrate that Hydraulis outperforms existing systems by 1.32-2.66 times.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。