用4.1万条数据训练开源大模型，自动生成可复现的软件问题测试用例。
SWE-Tester: Training Open-Source LLMs for Issue Reproduction in Real-World Repositories
- 构建4.1万条高质量数据集，微调开源大模型生成问题复现代码。
- 在SWT-Bench验证集上成功率提升10%，代码覆盖率提升21%。
- 适合关注开源AI辅助测试、自动化调试的研究者与开发者。
软件测试对保障软件正确性和可靠性至关重要。从自然语言问题描述中自动生成功能复现测试用例，可提升开发者效率，推动“先写测试后写代码”的开发模式，并增强自动修复系统（如编程代理）的效果。现有方法多依赖闭源大模型，对开源模型探索有限。为此，我们提出SWE-Tester——一种训练开源大模型生成问题复现测试的全新流程。首先，从2600个开源GitHub仓库中构建包含4.1万实例的高质量训练数据集，并用于训练不同规模与架构的开源大模型。微调后的模型在SWT-Bench Verified基准上，成功率绝对提升最高达10%，代码覆盖率提升21%。进一步分析表明，推理时计算资源增加、数据量扩大及模型规模增大均带来持续改进。这些结果验证了该框架在推动开源大模型在该领域应用方面的有效性。
原文摘要 · Abstract (English)
Software testing is crucial for ensuring the correctness and reliability of software systems. Automated generation of issue reproduction tests from natural language issue descriptions enhances developer productivity by simplifying root cause analysis, promotes test-driven development -- "test first, write code later", and can be used for improving the effectiveness of automated issue resolution systems like coding agents. Existing methods proposed for this task predominantly rely on closed-source LLMs, with limited exploration of open models. To address this, we propose SWE-Tester -- a novel pipeline for training open-source LLMs to generate issue reproduction tests. First, we curate a high-quality training dataset of 41K instances from 2.6K open-source GitHub repositories and use it to train LLMs of varying sizes and families. The fine-tuned models achieve absolute improvements of up to 10\% in success rate and 21\% in change coverage on SWT-Bench Verified. Further analysis shows consistent improvements with increased inference-time compute, more data, and larger models. These results highlight the effectiveness of our framework for advancing open-source LLMs in this domain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。