现有故障诊断方法在新程序上准确率下降0.19，因模型依赖程序结构特征。
Evaluation-Strategy Gap in Fault Diagnosis of Deep Learning Programs
- 用5542条注入故障的训练日志，对比程序内与跨程序评估差异
- 跨程序评估时平衡准确率下降0.19，主要源于程序级特征结构
- 曲率特征对未知程序的不稳定检测有效，优化器特征仅在已见程序中有效
深度学习程序在训练过程中可能因多种原因失败，故障诊断是耗时费力的维护任务。现有诊断技术通常通过程序内交叉验证评估，但此类方法在涉及未见过的程序的部署场景中可能不适用。本文基于包含38个真实世界DL程序的5542条故障注入训练日志（DynFault），研究了这种评估策略差距。结果发现，现有诊断技术在程序内评估与整体程序保留评估之间存在0.19的平衡准确率差距。该差距源于特征中的程序级结构，因此我们分析了两种运行时特征集：曲率特征和优化器特征，并考察其在未见程序上的表现。结果显示，曲率特征在未见程序上对不稳定性检测有帮助，而优化器特征和激活特征仅在训练中见过的程序上有效。
原文摘要 · Abstract (English)
Deep Learning (DL) programs can fail during training for many reasons, and diagnosing the cause is a costly and time-consuming maintenance task. Techniques for diagnosing such failures are commonly assessed using within-program cross-validation, which may be inadequate for deployment settings involving previously unseen programs. It is therefore necessary to assess how performance differs across these settings and to identify the causes of any performance gap in established fault diagnosis techniques for DL. We investigate this gap using DynFault, a corpus of 5,542 fault-injected training traces from 38 real-world DL programs. We found a gap of 0.190 in balanced accuracy for existing fault diagnosis techniques between within-program evaluation and holding out whole programs. We also found the gap comes from program-level structure in the features, which led us to examine two runtime feature sets, curvature features and optimizer features, and their behavior on unseen programs. We found that curvature features are useful for instability detection on unseen programs, while optimizer and activation features help only on programs seen during training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。