对比了深度学习故障定位技术，发现现有方法在真实缺陷上表现差，但改进后效果显著。
An Empirical Study of Fault Localisation Techniques for Deep Learning
- 基于动态与静态分析对比多种故障定位方法
- 单个真实修复方案下召回率最高仅0.31，精度0.23
- 考虑等效修复方案后，最优工具召回率达0.61，适合开发者调试
随着深度神经网络（DNN）的广泛应用，开发人员在实现、测试和调试DNN时对辅助工具的需求日益增加。已有多种方法可自动分析并定位待测DNN中的潜在故障。本文评估并比较了现有的最先进故障定位技术，这些技术基于DNN的动态和静态分析。评估基于一个基准数据集，包含从漏洞报告平台获取的真实故障以及通过变异工具生成的故障模型。研究发现，使用单一特定的真值（如人工定义的修复方案）评估DNN故障定位工具时，性能极低（最高平均召回率为0.31，精度为0.23）。然而，当考虑针对给定故障DNN存在的其他等效修复方案时，性能显著提升。结果表明， extit{dfd} 是最有效的工具，在该基准上达到平均召回率0.61和精度0.41。
原文摘要 · Abstract (English)
With the increased popularity of Deep Neural Networks (DNNs), increases also the need for tools to assist developers in the DNN implementation, testing and debugging process. Several approaches have been proposed that automatically analyse and localise potential faults in DNNs under test. In this work, we evaluate and compare existing state-of-the-art fault localisation techniques, which operate based on both dynamic and static analysis of the DNN. The evaluation is performed on a benchmark consisting of both real faults obtained from bug reporting platforms and faulty models produced by a mutation tool. Our findings indicate that the usage of a single, specific ground truth (e.g., the human defined one) for the evaluation of DNN fault localisation tools results in pretty low performance (maximum average recall of 0.31 and precision of 0.23). However, such figures increase when considering alternative, equivalent patches that exist for a given faulty DNN. Results indicate that \dfd is the most effective tool, achieving an average recall of 0.61 and precision of 0.41 on our benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。