arXiv:2410.19794cs.CVcs.LG2024-10中稿 · to IEEE Transactio…

用生成对抗网络生成差异测试图像,发现模型间隐藏行为差异。

DiffGAN: A Test Generation Approach for Differential Testing of Deep Neural Networks for Image Analysis

  • 基于GAN和遗传算法生成能暴露模型差异的测试图像。
  • 生成触发输入数量是当前最佳方法的4倍,且更多样、更有效。
  • 无需模型内部信息,适合模型选择与集成场景。

深度神经网络(DNN)在各类应用中日益普及,但确保其可靠性仍具挑战性。当存在功能相似且精度相近的替代模型时,传统基于准确率的评估难以捕捉模型间的实际行为差异,尤其在测试数据有限的情况下,影响模型选型或融合效果。差分测试通过生成能暴露模型行为差异的测试输入来解决此问题。然而,现有方法多依赖模型内部结构或受限于初始种子输入。为此,我们提出DiffGAN,一种用于DNN图像分析模型差分测试的黑盒测试图像生成方法。DiffGAN利用生成对抗网络(GAN)与非支配排序遗传算法II(NSGA-II),结合自定义的多样性与差异性两个适应度函数,系统探索生成器输入空间,识别模型输出间的差异。该方法可生成具有特定特征的输入,有效触发模型行为分歧。我们在八个基于主流图像数据集训练的DNN模型对上评估了DiffGAN,结果表明其生成的触发输入数量为当前最优基线的四倍,在相同预算下表现出更高的多样性与有效性。此外,生成的测试样本提升了基于机器学习的模型选型机制准确率,可作为替代模型使用时的智能输出投票策略。

原文摘要 · Abstract (English)

Deep Neural Networks (DNNs) are increasingly deployed across applications. However, ensuring their reliability remains a challenge, and in many situations, alternative models with similar functionality and accuracy are available. Traditional accuracy-based evaluations often fail to capture behavioral differences between models, especially with limited test datasets, making it difficult to select or combine models effectively. Differential testing addresses this by generating test inputs that expose discrepancies in DNN model behavior. However, existing approaches face significant limitations: many rely on model internals or are constrained by available seed inputs. To address these challenges, we propose DiffGAN, a black-box test image generation approach for differential testing of DNN models. DiffGAN leverages a Generative Adversarial Network (GAN) and the Non-dominated Sorting Genetic Algorithm II to generate diverse and valid triggering inputs that reveal behavioral discrepancies between models. DiffGAN employs two custom fitness functions, focusing on diversity and divergence, to guide the exploration of the GAN input space and identify discrepancies between models' outputs. By strategically searching this space, DiffGAN generates inputs with specific features that trigger differences in model behavior. DiffGAN is black-box, making it applicable in more situations. We evaluate DiffGAN on eight DNN model pairs trained on widely used image datasets. Our results show DiffGAN significantly outperforms a SOTA baseline, generating four times more triggering inputs, with greater diversity and validity, within the same budget. Additionally, the generated inputs improve the accuracy of a machine learning-based model selection mechanism, which selects the best-performing model based on input characteristics and can serve as a smart output voting mechanism when using alternative models.

差分测试生成模型模型评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。