arXiv:2410.23578cs.SEcs.AI2024-10被引 5

用AI自动检测量子软件中的不稳定测试,提升开发效率。

Automating Quantum Software Maintenance: Flakiness Detection and Root Cause Analysis

  • 用Transformer和余弦相似度自动识别量子代码中的不稳定测试。
  • 新发现25个不稳定测试,数据集规模扩大54%,检测F1达0.8871。
  • 适合量子软件开发者与自动化测试研究者参考。

由于量子软件的复杂性和概率特性,不稳定测试(即无代码变更却时通时错)在通用软件工程及量子软件工程中均构成重大挑战,导致隐藏缺陷和开发资源浪费。本文旨在构建一个自动化框架,用于检测量子软件中的不稳定测试,并扩展量子不稳定测试数据集,克服人工方法的局限性。基于对14个量子软件仓库的前期人工分析,我们扩充了数据集,并利用Transformer模型与余弦相似度实现自动化检测。通过OpenAI GPT和Meta LLaMA系列大语言模型,评估其从代码和问题描述中检测与分类不稳定测试的能力。嵌入式Transformer表现良好:共识别出25个新的不稳定测试,使数据集规模扩大54%;顶级模型在检测任务上获得0.8871的F1分数,但在根因识别上仅达0.5839。本文提出一种基于机器学习的自动化不稳定测试检测框架,结果令人鼓舞,但表明在大型量子代码库中根因检测与分类仍需改进。未来工作将聚焦于优化检测技术并实现不稳定测试的自动修复。

原文摘要 · Abstract (English)

Flaky tests, which pass or fail inconsistently without code changes, are a major challenge in software engineering in general and in quantum software engineering in particular due to their complexity and probabilistic nature, leading to hidden issues and wasted developer effort. We aim to create an automated framework to detect flaky tests in quantum software and an extended dataset of quantum flaky tests, overcoming the limitations of manual methods. Building on prior manual analysis of 14 quantum software repositories, we expanded the dataset and automated flaky test detection using transformers and cosine similarity. We conducted experiments with Large Language Models (LLMs) from the OpenAI GPT and Meta LLaMA families to assess their ability to detect and classify flaky tests from code and issue descriptions. Embedding transformers proved effective: we identified 25 new flaky tests, expanding the dataset by 54%. Top LLMs achieved an F1-score of 0.8871 for flakiness detection but only 0.5839 for root cause identification. We introduced an automated flaky test detection framework using machine learning, showing promising results but highlighting the need for improved root cause detection and classification in large quantum codebases. Future work will focus on improving detection techniques and developing automatic flaky test fixes.

量子软件测试自动化AI检测不稳定测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。