用产品文档生成端到端测试代码，提升软件质量
A Study on the Improvement of Code Generation Quality Using Large Language Models Leveraging Product Documentation
- 用定制提示词让大模型理解产品文档意图
- 生成测试代码编译成功率高，功能覆盖优于需求文档
- 适合关注自动化测试与软件质量的开发者
大型语言模型（LLMs）在系统开发中的应用日益广泛，尤其在自动化代码和测试生成方面。尽管端到端（E2E）测试对保障应用质量至关重要，但现有研究多聚焦于单元测试，针对E2E测试代码生成的工作较少。本研究提出一种基于产品文档（如手册、常见问题、教程）使用大模型自动生成E2E测试代码的方法。该方法分两步：先解析文档意图，再生成可执行测试代码。在包含六个核心功能（如认证、个人资料、讨论）的Web应用上进行实验，结果显示，基于产品文档生成的测试代码具有较高的编译成功率和功能覆盖率，显著优于基于需求规格和用户故事生成的测试。这表明产品文档能有效提升E2E测试质量，进而改善整体软件质量。
原文摘要 · Abstract (English)
Research on using Large Language Models (LLMs) in system development is expanding, especially in automated code and test generation. While E2E testing is vital for ensuring application quality, most test generation research has focused on unit tests, with limited work on E2E test code. This study proposes a method for automatically generating E2E test code from product documentation such as manuals, FAQs, and tutorials using LLMs with tailored prompts. The two step process interprets documentation intent and produces executable test code. Experiments on a web app with six key features (e.g., authentication, profile, discussion) showed that tests generated from product documentation had high compilation success and functional coverage, outperforming those based on requirement specs and user stories. These findings highlight the potential of product documentation to improve E2E test quality and, by extension, software quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。