自动检测分布式训练中故障机器，3.6秒内响应，准确率超90%。
Minder: Faulty Machine Detection for Large-scale Distributed Model Training
- 通过分析监控指标异常模式，自动识别故障机器。
- 平均3.6秒内发现故障，精度0.904，F1-score 0.893。
- 已上线一年，适用于千机级分布式训练场景。
大规模分布式模型训练需在数千台机器上同时进行，机器意外故障时故障检测至关重要。据我们经验，每日平均每训练任务会遭遇两次故障，可能导致数小时停摆。为克服人工排查耗时耗力的缺陷，我们提出Minder——一种面向分布式训练任务的自动化故障机器检测系统。其核心思路是自动高效识别故障前持续存在的特定监控指标异常模式。Minder已在生产环境部署超过一年，每日监控涉及上千台机器的分布式训练任务。在真实故障场景中，Minder平均可在3.6秒内准确响应，精度达0.904，F1-score为0.893。
原文摘要 · Abstract (English)
Large-scale distributed model training requires simultaneous training on up to thousands of machines. Faulty machine detection is critical when an unexpected fault occurs in a machine. From our experience, a training task can encounter two faults per day on average, possibly leading to a halt for hours. To address the drawbacks of the time-consuming and labor-intensive manual scrutiny, we propose Minder, an automatic faulty machine detector for distributed training tasks. The key idea of Minder is to automatically and efficiently detect faulty distinctive monitoring metric patterns, which could last for a period before the entire training task comes to a halt. Minder has been deployed in our production environment for over one year, monitoring daily distributed training tasks where each involves up to thousands of machines. In our real-world fault detection scenarios, Minder can accurately and efficiently react to faults within 3.6 seconds on average, with a precision of 0.904 and F1-score of 0.893.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。