用检索增强生成提升机器学习代码测试覆盖率,效果显著。
Retrieval-Augmented Test Generation: How Far Are We?
- 从官方文档、GitHub问题、StackOverflow中检索知识增强测试生成
- 平均提升6.5%代码覆盖率,GitHub问题贡献最大
- 生成的测试可发现28个新bug,适合开发与质量保障人员
检索增强生成(RAG)在软件工程中取得进展,但在单元测试生成领域仍研究不足。本文探究基于RAG的机器学习/深度学习API单元测试生成效果,并分析不同知识源的影响。研究覆盖五个主流Python ML/DL库:TensorFlow、PyTorch、Scikit-learn、Google JAX和XGBoost,聚焦高频使用API。采用四种先进大模型(GPT-3.5-Turbo、GPT-4o、Mistral MoE 8x22B、Llama 3.1 405B),对比三种策略:基础指令提示、基础RAG和API级RAG。评估指标包括语法正确性、动态正确性和行覆盖率。结果显示,尽管正确性未提升,但RAG使行覆盖率平均提高6.5%。其中,GitHub问题提供的边缘案例带来最佳提升效果。生成的测试共发现28个新缺陷:24个已提交开发者,10个获确认,4个被拒绝,10个待确认。研究揭示了针对性知识源对提升测试覆盖率的潜力,未来需优化检索技术以识别具有独特程序状态的文档。
原文摘要 · Abstract (English)
Retrieval Augmented Generation (RAG) has advanced software engineering tasks but remains underexplored in unit test generation. To bridge this gap, we investigate the efficacy of RAG-based unit test generation for machine learning (ML/DL) APIs and analyze the impact of different knowledge sources on their effectiveness. We examine three domain-specific sources for RAG: (1) API documentation (official guidelines), (2) GitHub issues (developer-reported resolutions), and (3) StackOverflow Q&As (community-driven solutions). Our study focuses on five widely used Python-based ML/DL libraries, TensorFlow, PyTorch, Scikit-learn, Google JAX, and XGBoost, targeting the most-used APIs. We evaluate four state-of-the-art LLMs -- GPT-3.5-Turbo, GPT-4o, Mistral MoE 8x22B, and Llama 3.1 405B -- across three strategies: basic instruction prompting, Basic RAG, and API-level RAG. Quantitatively, we assess syntactical and dynamic correctness and line coverage. While RAG does not enhance correctness, RAG improves line coverage by 6.5% on average. We found that GitHub issues result in the best improvement in line coverage by providing edge cases from various issues. We also found that these generated unit tests can help detect new bugs. Specifically, 28 bugs were detected, 24 unique bugs were reported to developers, ten were confirmed, four were rejected, and ten are awaiting developers' confirmation. Our findings highlight RAG's potential in unit test generation for improving test coverage with well-targeted knowledge sources. Future work should focus on retrieval techniques that identify documents with unique program states to optimize RAG-based unit test generation further.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。