无需修改代码即可自动诊断GPU训练失败原因并发送带日志的报警邮件。
GPUAlert: A Zero-Instrumentation Process-Boundary Monitor for Diagnosing GPU Training-Job Failures

- 通过进程边界监控,在不改代码前提下捕获训练失败
- 12类故障诊断准确率达99.7%,远超传统方法
- 适合大规模训练集群运维人员快速定位问题
GPU训练任务经常失败,大型生产集群中约有40%的任务会失败,但运维人员通常要数小时后才能发现。实验追踪工具需修改训练脚本且依赖云端连接;调度器邮件通知仅提供一行状态,无具体原因和日志。GPUAlert是一个命令行封装工具,可在不修改训练命令的前提下,在进程边界进行监控,并在任务完成后自动发送结构化通知,包含分类后的失败原因、持久化日志和输出产物。该工具基于三大可靠性原语:预启动日志保障(任务前确定日志持久存储位置)、通知器隔离(包装器退出码仅由子进程状态决定,不受邮件发送影响)、非静默产物预算(限制附件大小,从不无声丢弃输出)。我们发布了包含474条标注日志的语料库,涵盖15类失败模式,以及可复现的评估工具包。在12类硬件可复现的故障中,有序规则分类器达到0.997宏F1,远超无序关键词匹配的0.830和仅检查退出码的0.133。包装器开销恒定约每任务3ms;预启动保障确保日志在壳重定向失效时仍能保存;在所有15类故障中,即使SMTP代理不可达,包装器仍保持返回子进程原始退出码。
原文摘要 · Abstract (English)
GPU training jobs fail often, roughly two in five on large production clusters, yet the operator typically learns of a failure only by reconnecting hours later. Experiment trackers require editing the training script and maintaining a cloud connection; the scheduler's mail hook delivers a single status line with no cause and no logs. GPUAlert is a command-line wrapper that monitors any training command at the process boundary, and with no change to that command, emails a structured notification on completion carrying a classified failure cause, durable logs, and output artifacts. The tool is organized around three reliability primitives: a pre-launch log guarantee that establishes the durable destination before the child process can crash, notifier isolation that makes the wrapper's exit code a pure function of the child's status regardless of whether the email succeeds, and a non-silent artifact budget that bounds attachment size without ever dropping output silently. We release a labelled corpus of 474 GPU training logs across 15 failure classes and a reproducible evaluation harness. On the twelve hardware-reproduced classes, the ordered-rule classifier reaches 0.997 macro-F1, against 0.830 for unordered keyword matching and 0.133 for exit-code inspection. Wrapper overhead is a constant approximately 3ms per job; the pre-launch guarantee preserves a log where a shell redirect yields nothing; and across all 15 failure modes the wrapper returns the child's exit code unchanged even when the SMTP relay is unreachable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。