arXiv:2502.02910cs.SEcs.LG2025-02被引 3

用扩散模型生成数据替代训练集,实现无需原始数据的神经网络测试评估。

DANDI: Diffusion as Normative Distribution for Deep Neural Network Input

  • 用Stable Diffusion生成模拟训练数据,替代真实训练集计算输入异常度。
  • 在ImageNet-1K和CIFAR-10上与真实数据相关性达0.852和0.881。
  • 适用于无法获取训练数据的实际场景,提升神经网络测试可操作性。

意外充分性(SA)作为测试充分性度量,能有效引导软件工程师发现深度神经网络(DNN)中更可能暴露异常行为的输入。直观上,SA是一种分布外度量,用于量化输入与训练数据之间的差异:若新输入与训练时见过的数据差异显著,DNN更可能对输入表现出意外行为。尽管SA被广泛用作测试优先级方法,其主要缺陷在于计算需访问训练数据集,而现实中常受限于数据隐私或权限。本文提出DANDI,利用Stable Diffusion生成代理输入分布,从而在无需原始训练数据的情况下计算SA值。对图像分类器在CIFAR10和ImageNet-1K上的实证评估显示,基于合成数据计算的SA值与真实训练数据计算结果高度相关,相关系数分别为0.852(ImageNet-1K)和0.881(CIFAR-10)。此外,使用DANDI计算的SA值在对抗DeepMutation变异的DNN模型测试中,能同样有效优先排序输入。我们认为DANDI显著提升了SA在实际DNN测试中的可用性。

原文摘要 · Abstract (English)

Surprise Adequacy (SA) has been widely studied as a test adequacy metric that can effectively guide software engineers towards inputs that are more likely to reveal unexpected behaviour of Deep Neural Networks (DNNs). Intuitively, SA is an out-of-distribution metric that quantifies the dissimilarity between the given input and the training data: if a new input is very different from those seen during training, the DNN is more likely to behave unexpectedly against the input. While SA has been widely adopted as a test prioritization method, its major weakness is the fact that the computation of the metric requires access to the training dataset, which is often not allowed in real-world use cases. We present DANDI, a technique that generates a surrogate input distribution using Stable Diffusion to compute SA values without requiring the original training data. An empirical evaluation of DANDI applied to image classifiers for CIFAR10 and ImageNet-1K shows that SA values computed against synthetic data are highly correlated with the values computed against the training data, with Spearman Rank correlation value of 0.852 for ImageNet-1K and 0.881 for CIFAR-10. Further, we show that SA value computed by DANDI achieves can prioritize inputs as effectively as those computed using the training data, when testing DNN models mutated by DeepMutation. We believe that DANDI can significantly improve the usability of SA for practical DNN testing.

神经网络测试扩散模型输入评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。