通过MRC与SRv6实现超大规模AI训练的抗中断网络架构
Resilient AI Supercomputer Networking using MRC and SRv6
- 采用MRC多路径传输协议动态负载均衡，避免流量冲突
- 结合多平面Clos拓扑支持超10万GPU集群，提升冗余性
- 基于SRv6静态路由实现故障自愈，保障训练连续性
在超大规模同步预训练任务中，尾部延迟是性能瓶颈。本文提出三重方案：(1) 新型基于RDMA的传输协议MRC，通过多路径分发并主动负载均衡，消除流碰撞问题；(2) 采用多平面Clos拓扑，在保持高交换机端口密度的同时提供冗余，使超过10万GPU的训练集群可仅用两层拓扑构建；(3) 使用静态源路由技术SRv6，赋予MRC自主绕过故障的能力。我们在OpenAI和微软最大训练集群中部署MRC与静态SRv6路由，成功用于最新前沿模型的训练。实验表明，MRC可使训练任务在遭遇大量网络故障时仍能持续运行，此前此类故障将导致训练中断。
原文摘要 · Abstract (English)
Tail latency dominates the performance of synchronous pretraining jobs when running at very large scales. We describe a three-pronged approach: (1) a new RDMA-based transport protocol, MRC, sprays across many paths and actively load-balances between them, eliminating the issue of flow collisions (2) the use of multi-plane Clos topologies to get the benefits of high switch radix and redundancy, allowing training clusters well over 100K GPUs to be built as two-tier topologies while increasing physical redundancy, and (3) the use of static source-routing using SRv6 to allow MRC the freedom to bypass failures by itself. We describe our experiences running MRC and static SRv6 routing in production in OpenAI and Microsoft's largest training clusters, where it has been used to train the latest frontier models. We demonstrate how MRC allows AI training jobs to ride out many network failures that previously would have interrupted training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。