用自动检测机制提前发现深度学习训练中的隐藏错误。
Training with Confidence: Catching Silent Errors in Deep Learning Training with Automated Proactive Checks
- 基于深度学习训练特点自动推导不变性规则。
- 单次迭代内成功检测18个真实错误，发现6个库级漏洞。
- 适合调试深度学习训练过程的研究者和开发者。
深度学习训练过程复杂，易产生难以发现的隐藏错误。本文提出TRAINCHECK框架，采用主动检查策略，自动推导适用于深度学习训练的不变性规则，实现在训练过程中提前检测隐藏错误，并提供调试支持。为评估该框架，我们复现了20个具有不同根本原因的真实世界隐藏错误。TRAINCHECK在单次训练迭代内成功检测到18个错误，并发现6个流行训练库中存在的导致隐藏错误的未知缺陷。
原文摘要 · Abstract (English)
Training deep learning (DL) models is a complex process, making it prone to silent errors that are challenging to detect and diagnose. This paper presents TRAINCHECK, a framework that takes a proactive checking approach to address silent training errors. TRAINCHECK automatically infers invariants tailored for DL training. It uses these invariants to proactively detect silent errors during the training process while providing debugging help. To evaluate TRAINCHECK, we reproduce 20 real-world silent training errors with diverse root causes. TRAINCHECK successfully detects 18 errors within a single training iteration. It also uncovers 6 unknown bugs in popular training libraries that lead to silent errors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。