构建可执行的测试与代码协同演化基准，评估自动化工具真实理解变更能力。
TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution
- 基于真实提交历史构建可运行的测试生成与更新任务
- 测试通过率最高达77.5%，更新成功率74.6%，近期任务表现下降明显
- 支持按训练截止时间筛选任务，防止数据泄露，适合评估AI测试代理
软件测试与代码协同演化：代码变更后应伴随新或更新的测试以记录新行为。现有测试生成与更新基准常将测试与代码变更分离，依赖静态元数据，无法验证测试是否可执行或语义上关联变更。这使得难以评估测试自动化代理是否真正理解代码变更如何影响测试集。我们提出TestEvo-Bench，一个从软件仓库中挖掘的测试与代码协同演化任务基准，包含两个赛道：测试生成（代理需编写新测试以捕获新行为），测试更新（代理需调整失败测试以适配变更后的代码）。每个任务基于真实提交历史，附带环境配置，支持执行级指标如通过率、覆盖率和突变分数。TestEvo-Bench为实时基准：每项任务记录测试与代码变更时间戳，新任务由自动化流水线定期挖掘，评估可限定在模型训练截止时间之后的任务，降低数据泄露风险。当前快照包含746个测试生成任务与509个测试更新任务，源自152个开源Java项目中的59,950条候选协同演化记录。我们测试了四种先进代理，结合强大工具（Claude Code、Gemini CLI、SWE-Agent）与强基座模型（Claude Opus 4.7、Gemini 3.1 Pro）。结果表明，其测试生成成功率最高达77.5%，测试更新成功率为74.6%。然而，在最新任务上成功率显著下降，且在单任务成本受限时表现急剧下滑。
原文摘要 · Abstract (English)
Software tests and code evolve together: a code change should be followed by new or updated tests that record the new software behavior. Yet existing test generation and update benchmarks often isolate the test from the code change, and rely on static metadata that does not verify whether a test is executable or semantically tied to the code change. This makes it difficult to evaluate whether a test automation agent understands how a code change should propagate into the test suite. We introduce TestEvo-Bench, a benchmark of test and code co-evolution tasks mined from software repositories, with two tracks: in test generation, the agent shall write new tests to capture the new software behavior; in test update, the agent shall adapt failing existing tests to the changed software behavior. Each task is anchored to a real commit history and packaged with environment configuration to support execution-grounded metrics such as pass rate, coverage, and mutation score. TestEvo-Bench is also a live benchmark: each task records the timestamp of the test and code changes, and new tasks are periodically mined by our automated pipeline, so evaluation can be restricted to tasks postdating a model's training cutoff to reduce data leakage risk. The current snapshot contains 746 test generation and 509 test update tasks, curated from 59,950 candidate co-evolution records across 152 open-source Java projects. We experiment with four state-of-the-art agents that combine strong harnesses (Claude Code, Gemini CLI, and SWE-Agent) with strong foundation models (Claude Opus 4.7 and Gemini 3.1 Pro). Results show that they achieve up to 77.5% success rate on test generation and 74.6% on test update. However, success rate is materially lower on the most recent benchmark tasks and drops significantly under limited per-task cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。