Mycroft可快速定位大模型训练中通信故障根源
Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Training
- 通过追踪通信状态与依赖关系分析故障
- 90%场景15秒内发现异常，60%在20秒内定位根因
- 适合大规模分布式训练调试人员使用
可靠性对保障大模型训练效率至关重要。然而，许多现实中的可靠性问题难以解决，导致资源浪费和模型性能下降。当前的集体通信库如同黑箱，隐藏了进行有效根因分析所需的关键信息。我们提出Mycroft，一个轻量级的分布式追踪与根因分析系统，旨在解决集体通信中以往隐藏的可靠性问题。Mycroft的核心思想是追踪集体通信状态，并利用内部控制与数据依赖关系来化解大模型训练中的可靠性问题。Mycroft已在字节跳动部署超过六个月，用于实时调试集体通信相关问题。在90%的案例中，能在15秒内检测到异常；在60%的案例中，可在20秒内识别出根本原因。我们还进行了广泛的故障注入实验，以验证Mycroft的能力与效率。
原文摘要 · Abstract (English)
Reliability is essential for ensuring efficiency in LLM training. However, many real-world reliability issues remain difficult to resolve, resulting in wasted resources and degraded model performance. Unfortunately, today's collective communication libraries operate as black boxes, hiding critical information needed for effective root cause analysis. We propose Mycroft, a lightweight distributed tracing and root cause analysis system designed to address previously hidden reliability issues in collective communication. Mycroft's key idea is to trace collective communication states and leverage internal control and data dependencies to resolve reliability problems in LLM training. Mycroft has been deployed at ByteDance for over six months to debug collective communication related issues at runtime. It detected anomalies within 15 seconds in 90% of cases and identified the root cause within 20 seconds in 60% of cases. We also conducted extensive fault injection experiments to demonstrate Mycroft's capability and efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。