自动为不同大模型定制测试用例生成提示词,提升测试质量
The Prompt Alchemist: Automated LLM-Tailored Prompt Optimization for Test Case Generation
- 设计自动化优化框架,为每类大模型生成专属提示词
- 在多个开源项目上使测试用例生成准确率提升37%
- 融合领域知识与多样化搜索策略,避免重复错误
测试用例对保障软件可靠性至关重要。近期研究证明大语言模型(LLMs)可生成有效测试用例,但现有方法多依赖人工编写的简单提示词,导致性能受限。不同大模型对提示词的敏感度不同,统一提示词难以发挥最优效果。面对提示词形式繁多,自动发现适配各模型的最佳提示面临挑战。现有自动提示优化方法在自然语言处理中虽有应用,但在测试用例生成任务中表现不佳:一是仅通过组合和变异现有提示进行迭代优化,缺乏有效引导,导致提示多样性差且反复出现相同错误;二是提示普遍缺乏领域上下文知识,限制了大模型在任务中的表现。
原文摘要 · Abstract (English)
Test cases are essential for validating the reliability and quality of software applications. Recent studies have demonstrated the capability of Large Language Models (LLMs) to generate useful test cases for given source code. However, the existing work primarily relies on human-written plain prompts, which often leads to suboptimal results since the performance of LLMs can be highly influenced by the prompts. Moreover, these approaches use the same prompt for all LLMs, overlooking the fact that different LLMs might be best suited to different prompts. Given the wide variety of possible prompt formulations, automatically discovering the optimal prompt for each LLM presents a significant challenge. Although there are methods on automated prompt optimization in the natural language processing field, they are hard to produce effective prompts for the test case generation task. First, the methods iteratively optimize prompts by simply combining and mutating existing ones without proper guidance, resulting in prompts that lack diversity and tend to repeat the same errors in the generated test cases. Second, the prompts are generally lack of domain contextual knowledge, limiting LLMs' performance in the task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。