用进化算法高效生成可复现的缺陷测试,成本仅为传统方法的几分之一。
EvoOtter: Evolutionary Reproduction Test Generator

- 通过批量交叉和规则突变降低大模型调用开销,结合逐轮淘汰控制测试成本。
- 在10个真实项目上生成的测试用例复现成功率达92.3%,优于现有方法。
- 适合需要快速验证缺陷修复效果的开发者或自动化测试团队使用。
在修复问题前,生成能复现该问题的测试用例(BRT)很有价值。但生成高质量的BRT极具挑战性,因问题描述常不规范,难以判断候选测试是否真正符合问题原因。现有方法依赖大语言模型生成大量BRT与补丁,并通过执行反馈筛选优化,但代价高昂且反馈不可靠。本文提出新方法EvoOtter,采用进化编程策略,通过逐轮淘汰机制控制测试执行成本;同时利用单次调用完成整代的批量交叉、规则代码突变,并设计针对BRT的新型适应度评分。实验表明,EvoOtter在10个真实项目上以不足先前方法1/5的成本,达到92.3%的复现成功率,显著优于现有技术。本工作揭示了如何高效结合进化编程与大语言模型用于软件工程任务。
原文摘要 · Abstract (English)
Before fixing an issue, it is useful to first reproduce it by generating a bug reproduction test (BRT). However, generating a BRT is itself a challenging task, because issue descriptions tend to be informal, making it difficult to determine whether a candidate BRT indeed fails for the reason in the issue. Prior work has attempted to tackle this problem via inference scaling, using large language models to generate many BRTs and patches, then using execution feedback to select and improve them. Unfortunately, this is expensive and the feedback is unreliable. This paper explores evolutionary programming for BRT generation to sharpen the feedback, while enhancing evolutionary programming to keep costs in check. Our new approach, EvoOtter, controls test execution costs via successive halving. Furthermore, it controls LLM costs via batched crossover for an entire generation in a single LLM call, as well as via rule-based code mutations, with a new fitness score tailored for BRTs. As a result, EvoOtter generates state-of-the-art quality BRTs at the fraction of the cost of prior inference-scaling approaches to this problem. More broadly, this paper points at how to efficiently and effectively combine evolutionary programming with large language models for software engineering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。