用大模型生成测试验证步骤，效果参差但有潜力。
On the Effectiveness of LLMs for Manual Test Verifications
- 用8个大模型生成3.7万条测试验证，对比原版相似度。
- 开源模型如Mistral-7B表现接近闭源模型，但整体准确率仅40%。
- 适合想尝试AI辅助测试的开发者，需警惕幻觉问题。
手动测试对发现自动化测试遗漏的问题至关重要，但编写准确的验证步骤困难。本研究探索使用大语言模型（LLMs）生成手动测试验证步骤。通过两项互补的探索性研究：第一项使用2个闭源和6个开源LLMs生成验证，并评估其与原始验证的相似性；第二项邀请软件测试专家评估生成验证与其原始版本的一致性。结果表明，开源模型Mistral-7B和Phi-3-mini-4k在生成质量上与闭源模型Gemini-1.5-flash和GPT-3.5-turbo相当。然而，专业测试人员的认同率仅为略高于40%，显示出潜力与不足并存。部分生成验证优于原版，但也存在严重偏离预期的AI幻觉现象。研究贡献包括构建了一个由8个不同模型生成的37,040条测试验证数据集。尽管模型具备潜力，但40%左右的同意率表明仍需改进生成内容的准确性、相关性和清晰度，以提升实际测试场景中的可靠性。
原文摘要 · Abstract (English)
Background: Manual testing is vital for detecting issues missed by automated tests, but specifying accurate verifications is challenging. Aims: This study aims to explore the use of Large Language Models (LLMs) to produce verifications for manual tests. Method: We conducted two independent and complementary exploratory studies. The first study involved using 2 closed-source and 6 open-source LLMs to generate verifications for manual test steps and evaluate their similarity to original verifications. The second study involved recruiting software testing professionals to assess their perception and agreement with the generated verifications compared to the original ones. Results: The open-source models Mistral-7B and Phi-3-mini-4k demonstrated effectiveness and consistency comparable to closed-source models like Gemini-1.5-flash and GPT-3.5-turbo in generating manual test verifications. However, the agreement level among professional testers was slightly above 40%, indicating both promise and room for improvement. While some LLM-generated verifications were considered better than the originals, there were also concerns about AI hallucinations, where verifications significantly deviated from expectations. Conclusion: We contributed by generating a dataset of 37,040 test verifications using 8 different LLMs. Although the models show potential, the relatively modest 40% agreement level highlights the need for further refinement. Enhancing the accuracy, relevance, and clarity of the generated verifications is crucial to ensure greater reliability in real-world testing scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。