动态调整大模型训练并行策略，实现亚秒级无损切换。
DynaTrain: Fast Online Parallelism Switching for Elastic LLM Training
- 用虚拟参数空间统一分布式状态，实现任意并行配置的快速映射。
- 70B模型切换耗时不足2秒，235B MoE模型仅需4.36秒。
- 适合需要弹性资源调度的大规模模型训练场景。
现代大语言模型训练具有高度动态性：资源波动、强化学习微调阶段变化及集群弹性不断改变最优并行布局，给基于静态执行模型的现有训练框架带来挑战。我们提出DynaTrain，一个支持亚秒级在线重构的分布式训练系统，可实现任意多维并行性的快速切换。其核心是虚拟参数空间（VPS）抽象，将所有分布式训练状态统一到单一逻辑坐标系中，使任何并行配置变为确定性映射，将复杂迁移简化为可管理的几何交集。在此基础上，状态路由与迁移层在内存感知、无死锁的调度下执行本地秩间传输，弹性设备管理器则将新拓扑构建与训练过程重叠，隐藏拓扑变更开销。在高达2350亿参数的密集模型和MoE模型上，DynaTrain将700亿参数模型的重新配置时间控制在2秒以内，2350亿参数的MoE模型仅需4.36秒，较最先进基于检查点和弹性系统快达三个数量级，且保证正确性。
原文摘要 · Abstract (English)
Modern large language model (LLM) training is inherently dynamic: resource fluctuations, RLHF phase shifts, and cluster elasticity continually reshape the optimal parallelism layout, posing a significant challenge to existing training frameworks built around a static execution model. We present DynaTrain, a distributed training system for sub-second, online reconfiguration across arbitrary multi-dimensional parallelism. At its core, we propose a Virtual Parameter Space (VPS) abstraction that unifies all distributed training states under one logical coordinate space, turning any parallelism configuration into a deterministic mapping and collapsing complex transition into manageable geometric intersections. On top of VPS, a state routing-and-transition layer executes rank-local transfers under a memory-aware, deadlock-free schedule, and an Elastic Device Manager overlaps new-world construction with ongoing training to mask topology-change cost. On dense and MoE models up to 235B parameters, DynaTrain reconfigures a 70B dense model in under 2s and a 235B MoE model in 4.36s, outperforming state-of-the-art checkpoint-based and elastic systems by up to three orders of magnitude while preserving correctness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。