arXiv:2505.12815cs.DCcs.AI2025-05

提出自愈与弹性伸缩的分布式训练系统,应对跨机构集群的节点频繁变动。

Learning In Chaos: Efficient Autoscaling and Self-Healing for Multi-Party Distributed Training

  • 通过多邻居状态复制和模型分片加速自动扩缩容
  • 在真实网络环境下实现20ms内完成扩缩容与链路故障处理
  • 适合需要自主治理的跨机构联合训练场景

跨区域广域网中的多参与方分布式训练常因节点和链路频繁变动而中断。现有基于检查点恢复和云中心化扩缩容的方法反应迟缓,且依赖集中控制,与机构自由加入退出的自治架构不匹配。本文提出Chaos系统,支持自愈与自适应扩缩容,在节点变动下保持训练稳定与弹性。通过多邻居状态复制与模型分片加速扩缩容,并将分片与分配问题建模为考虑广域网异构性的混合整数非线性规划(MINLP),利用可除性链的单调性分析将其简化为可求解的混合整数线性规划(MILP),进而设计出遵循最优规则的贪心算法,可在多项式时间内获得最优解。Chaos采用集群监控器实时感知资源与拓扑变化,通过节点间协商协议处理扩缩容事件,实现完全去中心化的自治扩缩容。实验表明,Chaos的扩容延迟显著低于Pollux、Elan和传统扩缩容方案,在20ms内完成缩容、链路连接与断开事件处理,同时拥有最低空闲时间,随集群规模增长展现出更优的资源利用率与可扩展性。

原文摘要 · Abstract (English)

Node and link churn in multi-party, cross-region clusters over wide-area networks (WANs) often disrupts distributed training. However, checkpoint-based recovery and cloud-centric autoscaling react slowly and assume centralized control, which is misaligned with the self-governed setup where institutions can freely join and leave. This paper proposes Chaos, a multi-party distributed training system with self-healing and autoscaling, enabling robust and elastic training under churn. It speeds up autoscaling via multi-neighbor state replication and model sharding. We formalize the sharding and assignment as a MINLP that captures WAN heterogeneity, and reduce it to a tractable MILP by analyzing its monotonicity on a divisibility chain. By establishing an equivalence, we derive a greedy algorithm that follows optimality rules and yields the optimal solution in polynomial time. Chaos uses a cluster monitor to track resource and topology changes, and handles scaling events through peer negotiation protocols, enabling fully self-governed autoscaling among institutions. Experiments show that Chaos has substantially lower scale-out delay than Pollux, Elan, and Autoscaling, and handles scale-in, connect-link, and disconnect-link events within 20ms. It also delivers the lowest idle time, showing superior resource use and scalability as the cluster grows.

分布式训练弹性扩缩容自愈系统跨机构协作

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。