用生成式AI生成测试图像，评估深度学习模型鲁棒性。
Benchmarking Generative AI Models for Deep Learning Test Input Generation
- 对比VAE、GAN、扩散模型生成测试输入的性能
- 扩散模型在ImageNet上生成更多有效误分类样本
- 适合做深度学习模型可靠性测试的研究者
测试输入生成器（TIGs）对于评估深度学习图像分类器在训练和测试集之外输入上的正确预测能力至关重要。近年来生成式AI（GenAI）模型的发展使其成为生成和操纵合成图像的强大工具，但这些进展也带来了更高的复杂性和资源需求。本文对不同GenAI模型与TIGs的结合进行基准测试，评估其在生成测试图像时的有效性、效率及质量，包括领域有效性与标签保真度。我们通过一项实证研究，涵盖三种GenAI架构（VAEs、GANs、Diffusion Models）、五项复杂度递增的分类任务，以及364次人工评估。结果表明，对于MNIST等简单数据集，VAE等较简单架构已足够；而在ImageNet等特征丰富的数据集上，扩散模型能生成更多有效且引发误分类的输入，表现更优。
原文摘要 · Abstract (English)
Test Input Generators (TIGs) are crucial to assess the ability of Deep Learning (DL) image classifiers to provide correct predictions for inputs beyond their training and test sets. Recent advancements in Generative AI (GenAI) models have made them a powerful tool for creating and manipulating synthetic images, although these advancements also imply increased complexity and resource demands for training. In this work, we benchmark and combine different GenAI models with TIGs, assessing their effectiveness, efficiency, and quality of the generated test images, in terms of domain validity and label preservation. We conduct an empirical study involving three different GenAI architectures (VAEs, GANs, Diffusion Models), five classification tasks of increasing complexity, and 364 human evaluations. Our results show that simpler architectures, such as VAEs, are sufficient for less complex datasets like MNIST. However, when dealing with feature-rich datasets, such as ImageNet, more sophisticated architectures like Diffusion Models achieve superior performance by generating a higher number of valid, misclassification-inducing inputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。