评测大模型在自动维护单元测试中的表现,涵盖创建、修复和更新三类任务。
TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance
- 基于测试文件级的评估框架,支持真实代码仓库上下文下的自动化测试维护。
- 在1539个来自Python/Java/Go项目的场景中,顶尖模型仅带来有限效果提升。
- 开源工具包支持无参考评估,适合研究自动化测试与LLM结合的学者使用。
尽管大型语言模型在软件工程中展现潜力,但其在单元测试中的应用仍局限于孤立的测试生成或断言预测,忽视了测试套件维护这一更广泛的挑战。我们提出TAM-Eval(测试自动化维护评估)框架与基准,用于评估模型在创建、修复和更新测试套件三种核心维护场景中的表现。与以往局限于函数级别的任务不同,TAM-Eval以测试文件为单位进行评估,并在独立评估中保留完整仓库上下文,更贴近真实维护流程。该基准包含从Python、Java和Go项目中自动提取并验证的1,539个场景。TAM-Eval支持对原始LLM及智能体工作流的系统无关评估,采用基于测试通过率、代码覆盖率和变异测试的无参考协议。实证结果表明,当前最先进的大模型在真实测试维护流程中能力有限,仅带来微弱的效果提升。我们已将TAM-Eval开源,以支持未来自动化软件测试研究。数据与代码公开于https://github.com/trndcenter/TAM-Eval。
原文摘要 · Abstract (English)
While Large Language Models (LLMs) have shown promise in software engineering, their application to unit testing remains largely confined to isolated test generation or oracle prediction, neglecting the broader challenge of test suite maintenance. We introduce TAM-Eval (Test Automated Maintenance Evaluation), a framework and benchmark designed to evaluate model performance across three core test maintenance scenarios: creation, repair, and updating of test suites. Unlike prior work limited to function-level tasks, TAM-Eval operates at the test file level, while maintaining access to full repository context during isolated evaluation, better reflecting real-world maintenance workflows. Our benchmark comprises 1,539 automatically extracted and validated scenarios from Python, Java, and Go projects. TAM-Eval supports system-agnostic evaluation of both raw LLMs and agentic workflows, using a reference-free protocol based on test suite pass rate, code coverage, and mutation testing. Empirical results indicate that state-of-the-art LLMs have limited capabilities in realistic test maintenance processes and yield only marginal improvements in test effectiveness. We release TAM-Eval as an open-source framework to support future research in automated software testing. Our data and code are publicly available at https://github.com/trndcenter/TAM-Eval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。