用大模型生成可验证的测试用例,提前保障代码修复质量。
TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?
- 基于问题描述和代码库,用LLM生成符合‘先失败后通过’的测试用例。
- 在449个真实问题上,测试通过率高于现有最强方法,覆盖率达80%以上。
- 适合关注自动化测试与代码质量提升的研发团队使用。
测试驱动开发(TDD)强调先写测试再编码,能有效明确修复目标。尽管已有大量针对‘先编码后写测试’的自动化测试生成研究,但对TDD的自动化支持仍较少。理想的TDD测试应具备‘失败-通过’特性,并充分覆盖修复期间变更的代码。本文提出TDD-Bench Verified,一个从真实GitHub仓库中挖掘的高质量基准数据集,包含449个问题。该基准采用隔离运行机制,仅执行相关测试以实现精确覆盖率测量,并经人工与执行双重筛选。同时,本文提出Auto-TDD,一种基于LLM的方案,输入问题描述与修复前代码库,输出可用于验证修复结果的测试。评估显示,Auto-TDD在失败-通过率上优于现有最佳方法,且覆盖度达80%以上。本工作有望提升开发者效率并增强修复健壮性。
原文摘要 · Abstract (English)
Test-driven development (TDD) is the practice of writing tests first and coding later, and the proponents of TDD expound its numerous benefits. For instance, given an issue on a source code repository, tests can clarify the desired behavior among stake-holders before anyone writes code for the agreed-upon fix. Although there has been a lot of work on automated test generation for the practice "write code first, test later", there has been little such automation for TDD. Ideally, tests for TDD should be fail-to-pass (i.e., fail before the issue is resolved and pass after) and have good adequacy with respect to covering the code changed during issue resolution. This paper introduces TDD-Bench Verified, a high-quality benchmark suite of 449 issues mined from real-world GitHub code repositories. The benchmark's evaluation harness runs only relevant tests in isolation for simple yet accurate coverage measurements, and the benchmark's dataset is filtered both by human judges and by execution in the harness. This paper also presents Auto-TDD, an LLM-based solution that takes as input an issue description and a codebase (prior to issue resolution) and returns as output a test that can be used to validate the changes made for resolving the issue. Our evaluation shows that Auto-TDD yields a better fail-to-pass rate than the strongest prior work while also yielding high coverage adequacy. Overall, we hope that this work helps make developers more productive at resolving issues while simultaneously leading to more robust fixes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。