用大模型自动发现量子软件中的不稳定性测试并定位原因。
Automating Detection and Root-Cause Analysis of Flaky Tests in Quantum Software
- 用大模型和相似度算法自动化挖掘量子代码库中的潜在不稳定测试。
- 发现25个未知的不稳定性测试，数据集规模扩大54%，最佳模型F1达0.94。
- 适合量子软件工程师和测试自动化研究者使用，提升调试效率。
与经典软件类似，量子软件系统依赖自动化测试。然而其固有的概率输出使其易受量子不稳定性影响——即在无代码修改的情况下测试结果忽成忽败。这类不稳定性测试会掩盖真实缺陷，降低开发效率，但系统性检测与诊断工具仍十分有限。本文提出一种自动化流水线，用于检测量子软件仓库中的不稳定性测试及其相关代码提交，并支持根因分析。我们旨在扩展现有量子不稳定性测试数据集，并评估大语言模型（LLMs）在分类不稳定性及识别根因方面的能力。基于对14个量子软件仓库的前期人工分析，我们利用大模型与余弦相似度自动化发现更多不稳定性测试案例。进一步评估了来自OpenAI GPT、Meta LLaMA、Google Gemini和Anthropic Claude系列的多种大模型在从问题描述与代码上下文中分类不稳定性及识别根因的表现。分类性能通过标准指标如F1分数评估。通过该流水线，我们识别出25个此前未知的不稳定性测试，使原始数据集规模增加54%。表现最佳的模型为Google Gemini，其在不稳定性检测上取得0.9420的F1分数，在根因识别上达到0.9643，表明大模型可有效辅助处理不稳定性报告并理解其根本原因。所扩展的数据集与自动化流水线为量子软件工程社区提供了可复用资源。未来工作将聚焦于提升检测鲁棒性，并探索不稳定性测试的自动修复。
原文摘要 · Abstract (English)
Like classical software, quantum software systems rely on automated testing. However, their inherently probabilistic outputs make them susceptible to quantum flakiness -- tests that pass or fail inconsistently without code changes. Such quantum flaky tests can mask real defects and reduce developer productivity, yet systematic tooling for their detection and diagnosis remains limited. This paper presents an automated pipeline to detect flaky-test-related issues and pull requests in quantum software repositories and to support the identification of their root causes. We aim to expand an existing quantum flaky test dataset and evaluate the capability of Large Language Models (LLMs) for flakiness classification and root-cause identification. Building on a prior manual analysis of 14 quantum software repositories, we automate the discovery of additional flaky test cases using LLMs and cosine similarity. We further evaluate a variety of LLMs from OpenAI GPT, Meta LLaMA, Google Gemini, and Anthropic Claude suites for classifying flakiness and identifying root causes from issue descriptions and code context. Classification performance is assessed using standard performance metrics, including F1-score. Using our pipeline, we identify 25 previously unknown flaky tests, increasing the original dataset size by 54%. The best-performing model, Google Gemini, achieves an F1-score of 0.9420 for flakiness detection and 0.9643 for root-cause identification, demonstrating that LLMs can provide practical support for triaging flaky reports and understanding their underlying causes in quantum software. The expanded dataset and automated pipeline provide reusable artifacts for the quantum software engineering community. Future work will focus on improving detection robustness and exploring automated repair of quantum flaky tests.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。