用多智能体协作生成可执行的测试断言,提升代码测试准确性
Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesis
- 四类专家智能体协同设计断言,通过迭代自修正优化
- 在LiveCodeBench上将GPT-4.1-Mini的断言准确率从46.30%提至57.73%
- 显著提升漏洞检测与程序修复成功率,适合自动化测试研究者
非回归测试中的测试断言生成是软件工程长期面临的挑战,目标是为给定输入生成能准确判断被测函数(FUT)是否按预期行为的断言。本文提出Nexus,一种新型多智能体框架,通过一组专业化智能体协作,以结构化流程合成断言:首先由四类具有不同测试理念的专家智能体进行协作评审与优化;随后在安全沙箱中生成可能的FUT实现,并执行候选断言进行验证;对失败断言,系统自动启动调试与修正循环,基于运行时错误重构断言并重新验证。在七个多样化基准上的评估表明,Nexus持续且显著优于现有基线。例如,在LiveCodeBench上,GPT-4.1-Mini的断言准确率从46.30%提升至57.73%;在HumanEval上,生成断言的漏洞检测率从90.91%升至95.45%,自动化程序修复成功率从35.23%提升至69.32%。
原文摘要 · Abstract (English)
Test oracle generation in non-regression testing is a longstanding challenge in software engineering, where the goal is to produce oracles that can accurately determine whether a function under test (FUT) behaves as intended for a given input. In this paper, we introduce Nexus, a novel multi-agent framework to address this challenge. Nexus generates test oracles by leveraging a diverse set of specialized agents that synthesize test oracles through a structured process of deliberation, validation, and iterative self-refinement. During the deliberation phase, a panel of four specialist agents, each embodying a distinct testing philosophy, collaboratively critiques and refines an initial set of test oracles. Then, in the validation phase, Nexus generates a plausible candidate implementation of the FUT and executes the proposed oracles against it in a secure sandbox. For any oracle that fails this execution-based check, Nexus activates an automated selfrefinement loop, using the specific runtime error to debug and correct the oracle before re-validation. Our extensive evaluation on seven diverse benchmarks demonstrates that Nexus consistently and substantially outperforms state-of-theart baselines. For instance, Nexus improves the test-level oracle accuracy on the LiveCodeBench from 46.30% to 57.73% for GPT-4.1-Mini. The improved accuracy also significantly enhances downstream tasks: the bug detection rate of GPT4.1-Mini generated test oracles on HumanEval increases from 90.91% to 95.45% for Nexus compared to baselines, and the success rate of automated program repair improves from 35.23% to 69.32%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。