用问题自动生成可验证代码修复的测试，支持TDD和自动化编程。
Otter: Generating Tests from Issues to Validate SWE Patches
- 基于大模型+规则分析，自动生成与问题匹配的测试用例。
- 在TDD-Bench-Verified基准上超越现有系统，提升测试有效性。
- 适合想实践TDD或开发AI编程助手的研究者与工程师。
尽管已有大量工作致力于从代码生成测试，但针对从问题生成测试的研究仍较少。一个正确的测试必须能验证修复问题的代码补丁，而本文关注的是该补丁尚未存在的情况。这支持两大应用场景：一是支持TDD（测试驱动开发），即“先写测试，后写代码”，已被证实对人类开发者有益；二是验证软件工程代理（SWE agent）生成补丁的能力。本文提出TDD-Bench-Verified基准，以及基于大模型的Otter解决方案。Otter通过规则化分析增强大模型输出，并引入新颖的自我反思式动作规划器。实验表明，Otter在生成测试方面优于当前最先进系统，同时提升了生成补丁系统的性能。我们期望Otter能帮助开发者更高效地解决缺陷，推动更健壮、充分测试的代码产出。
原文摘要 · Abstract (English)
While there has been plenty of work on generating tests from existing code, there has been limited work on generating tests from issues. A correct test must validate the code patch that resolves the issue. This paper focuses on the scenario where that code patch does not yet exist. Doing so supports two major use-cases. First, it supports TDD (test-driven development), the discipline of "test first, write code later" that has well-documented benefits for human software engineers. Second, it also validates SWE (software engineering) agents, which generate code patches for resolving issues. This paper introduces TDD-Bench-Verified, a benchmark for generating tests from issues, and Otter, an LLM-based solution for this task. Otter augments LLMs with rule-based analysis to check and repair their outputs, and introduces a novel self-reflective action planner. Experiments show Otter outperforming state-of-the-art systems for generating tests from issues, in addition to enhancing systems that generate patches from issues. We hope that Otter helps make developers more productive at resolving issues and leads to more robust, well-tested code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。