用大模型自动执行任意项目的测试，成功率超六成。
You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects
- 基于大模型的智能代理，自主完成项目构建与测试脚本生成。
- 在50个不同项目中成功运行33个，结果偏差仅7.5%。
- 适合需要跨项目自动化测试的开发者与研究者使用。
执行项目测试套件在评估代码质量、验证变更和确保依赖兼容性方面至关重要。然而，由于项目使用不同的编程语言、软件生态、构建系统和测试框架，实现通用可靠的测试执行方法极具挑战。本文提出ExecutionAgent，一种基于大语言模型（LLM）的自动化技术，可从源码出发准备构建脚本并运行测试用例。该方法受人类开发者的启发，通过元提示获取最新技术指南，并根据前序步骤反馈迭代优化流程。我们在50个使用14种编程语言及多种构建与测试工具的开源项目上评估该方法，成功执行了33个项目的测试套件，测试结果与真实执行偏差仅为7.5%，性能优于先前最佳方法6.6倍。平均每次执行耗时74分钟，大模型成本为0.16美元。我们预计ExecutionAgent将成为开发者、自动化编程工具和研究者跨项目测试的重要助手。
原文摘要 · Abstract (English)
The ability to execute the test suite of a project is essential in many scenarios, e.g., to assess code quality and code coverage, to validate code changes made by developers or automated tools, and to ensure compatibility with dependencies. Despite its importance, executing the test suite of a project can be challenging in practice because different projects use different programming languages, software ecosystems, build systems, testing frameworks, and other tools. These challenges make it difficult to create a reliable, universal test execution method that works across different projects. This paper presents ExecutionAgent, an automated technique that prepares scripts for building an arbitrary project from source code and running its test cases. Inspired by the way a human developer would address this task, our approach is a large language model (LLM)-based agent that autonomously executes commands and interacts with the host system. The agent uses meta-prompting to gather guidelines on the latest technologies related to the given project, and it iteratively refines its process based on feedback from the previous steps. Our evaluation applies ExecutionAgent to 50 open-source projects that use 14 different programming languages and many different build and testing tools. The approach successfully executes the test suites of 33/50 projects, while matching the test results of ground truth test suite executions with a deviation of only 7.5%. These results improve over the best previously available technique by 6.6x. The costs imposed by the approach are reasonable, with an execution time of 74 minutes and LLM costs of USD 0.16, on average per project. We envision ExecutionAgent to serve as a valuable tool for developers, automated programming tools, and researchers that need to execute tests across a wide variety of projects.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。