arXiv:2410.11769cs.SEcs.AI2024-10中稿 · publication by Emp…被引 8

Pareto优化在测试中无法有效覆盖故障输入区域,效果不如随机搜索。

Can Search-Based Testing with Pareto Optimization Effectively Cover Failure-Revealing Test Inputs?

  • 用帕累托优化同时追求多个目标以发现故障
  • 两种算法在两个系统中覆盖故障输入的能力不如随机搜索
  • 适合关注测试覆盖率和故障多样性研究的开发者

基于搜索的软件测试(SBST)广泛用于测试输入空间庞大的复杂系统,如深度学习驱动(DL-enabled)系统。许多SBST方法采用帕累托优化,同时优化多个目标以揭示故障。然而,重要的是要确保识别出的故障在整个故障诱导区域内分布均匀,而非集中在子区域,这样才能保证故障语义多样,揭示多种根本原因。本文提出理论论证,说明基于帕累托优化的测试不足以覆盖搜索域中的故障诱导区域。通过将两种常用帕累托优化技术——NSGA-II(进化算法)和OMOPSO(群智能算法)——应用于两个DL-enabled系统(工业级自动代客泊车系统和手写数字分类系统),并使用“逆距离覆盖指标”衡量故障揭示测试输入在输入空间中的覆盖程度,结果表明:基于NSGA-II和OMOPSO的搜索在覆盖故障输入方面并不优于简单的随机搜索基线。本研究的复现代码已发布于GitHub仓库。

原文摘要 · Abstract (English)

Search-based software testing (SBST) is a widely adopted technique for testing complex systems with large input spaces, such as Deep Learning-enabled (DL-enabled) systems. Many SBST techniques focus on Pareto-based optimization, where multiple objectives are optimized in parallel to reveal failures. However, it is important to ensure that identified failures are spread throughout the entire failure-inducing area of a search domain and not clustered in a sub-region. This ensures that identified failures are semantically diverse and reveal a wide range of underlying causes. In this paper, we present a theoretical argument explaining why testing based on Pareto optimization is inadequate for covering failure-inducing areas within a search domain. We support our argument with empirical results obtained by applying two widely used types of Pareto-based optimization techniques, namely NSGA-II (an evolutionary algorithm) and OMOPSO (a swarm-based Pareto-optimization algorithm), to two DL-enabled systems: an industrial Automated Valet Parking (AVP) system and a system for classifying handwritten digits. We measure the coverage of failure-revealing test inputs in the input space using a metric that we refer to as the Coverage Inverted Distance quality indicator. Our results show that NSGA-II-based search and OMOPSO are not more effective than a naïve random search baseline in covering test inputs that reveal failures. The replication package for this study is available in a GitHub repository.

软件测试帕累托优化故障覆盖SBST

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。