veScale让分布式训练像单机一样简单且高效。
veScale: Consistent and Efficient Tensor Programming with Eager-Mode SPMD
- 采用即时执行的SPMD模式，统一编程接口
- 相比现有系统提升2.2倍速度，代码量减少78.4%
- 确保多设备结果与单机一致，适合大规模模型训练
大语言模型的快速扩展带来了日益复杂的分布式训练需求，如3D并行。这推动了向更简单、可调试的编程范式——单程序多数据（SPMD）的转变。然而，即时执行下的SPMD面临两大挑战：结果一致性与高可扩展性。本文提出veScale，一个完全采用即时执行SPMD范式的训练系统，旨在普及分布式张量编程。veScale通过引入与任意分片算子兼容的新型分布式随机数生成算法，解决了如PyTorch等系统中普遍存在的结果不一致问题。同时，通过降低PyTorch原语开销并优化通信效率，显著提升训练性能。实验表明，veScale相较当前最先进的TorchTitan系统最高提速2.2倍，代码复杂度降低78.4%，且保持与单设备等效的结果。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have scaled rapidly in size and complexity, requiring increasingly intricate parallelism for distributed training, such as 3D parallelism. This sophistication motivates a shift toward simpler, more debuggable programming paradigm like Single Program Multiple Data (SPMD). However, SPMD in eager execution introduces two key challenges: ensuring consistency with single-device execution and achieving high performance at scale. In this paper, we introduce veScale, an eager-mode training system that fully embraces SPMD paradigm to democratize distributed tensor programming. veScale addresses the prevalent issue of inconsistent results in systems like PyTorch by introducing a novel algorithm of distributed Random Number Generation (RNG) compatible with arbitrary sharded operators. veScale also significantly boosts training performance by reducing PyTorch primitive's overhead and improving communication efficiency. Evaluations show that veScale delivers up to 2.2x speedup over the state-of-the-art training systems, like TorchTitan, and cuts code complexity by 78.4%, while preserving single-device-equivalent results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。