TrainMover让大模型训练中断后20秒内恢复，大幅减少算力浪费。
TrainMover: An Interruption-Resilient Runtime for ML Training
- 用弹性机器和备用节点实现快速故障转移
- 1024卡规模下中断恢复仅需约20秒
- 适合大规模分布式训练团队使用
大规模机器学习训练常因软硬件异常、故障或管理事件而中断。现有检查点重启或运行时重配置方案存在长时间停机和性能下降问题。我们提出TrainMover，一种针对大模型训练的抗中断运行时系统，利用弹性与备用机器，在最小停机时间和零内存开销下应对中断。其核心包含三项技术：两阶段增量通信组建立、无通信沙箱预热，以及通用备用设计，支持任意角色故障恢复。评估显示，TrainMover在1024张GPU规模下处理各类中断时，停机时间稳定在约20秒。相比最优替代方案，预计在64K GPU规模下每周可减少55%的算力浪费，节省140万GPU小时。
原文摘要 · Abstract (English)
Large-scale ML training jobs are frequently interrupted by hardware and software anomalies, failures, and management events. Existing solutions like checkpoint-restart or runtime reconfiguration suffer from long downtimes and degraded performance. We present TrainMover, a resilient LLM training runtime that leverages elastic and standby machines to handle interruptions with minimal downtime and zero memory overhead. To achieve these goals, TrainMover introduces three key techniques: two-phase, delta-based communication group setup; communication-free sandboxed warmup; and general standby design that enables failure recovery from any role. Our evaluation shows that TrainMover consistently achieves around 20 seconds of downtime when handling various interruptions at the 1024-GPU scale. TrainMover is projected to reduce wasted GPU hours by 55% compared to the best alternative, saving 1.4 million GPU-hours per week at the 64K-GPU scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。