Echo可高精度模拟万卡训练，误差仅8%且2分钟内完成。
Echo: Simulating Distributed Training At Scale
- 基于设备级运行时追踪，单机还原千卡训练执行图。
- 无需高开销离散事件仿真，精准估算集体通信开销。
- 捕捉通信计算重叠导致的性能下降，适合大规模训练优化者。
仿真在枚举与外推方面具有独特价值，对管理大规模机器学习集群和分布式训练任务日益重要。本文提出Echo，解决大规模训练仿真中的三个关键挑战：(1) 以非侵入方式追踪每个设备的运行时工作负载，实现单设备还原1000卡训练的实际执行图；(2) 在不依赖高开销离散事件网络仿真的前提下，准确估算集体通信开销；(3) 考虑同一设备上通信与计算核函数重叠引发的计算延迟。Echo在96张H800 GPU、Megatron-LM采用3D并行的GPT-175B训练场景下，平均训练步误差仅8%，相较现有最优仿真器降低约3倍，且可在2分钟内完成仿真。
原文摘要 · Abstract (English)
Simulation offers unique values for both enumeration and extrapolation purposes, and is becoming increasingly important for managing the massive machine learning (ML) clusters and large-scale distributed training jobs. In this paper, we build Echo to tackle three key challenges in large-scale training simulation: (1) tracing the runtime training workloads at each device in an ex-situ fashion so we can use a single device to obtain the actual execution graphs of 1K-GPU training, (2) accurately estimating the collective communication without high overheads of discrete-event based network simulation, and (3) accounting for the interference-induced computation slowdown from overlapping communication and computation kernels on the same device. Echo delivers on average 8% error in training step -- roughly 3x lower than state-of-the-art simulators -- for GPT-175B on a 96-GPU H800 cluster with 3D parallelism on Megatron-LM under 2 minutes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。