多智能体协作提升软件测试准确率,降低幻觉风险。
Multi-Agent LLM Committees for Autonomous Software Beta Testing
- 多个视觉增强型大模型通过三轮投票达成测试共识。
- 任务成功率最高达91.7%,比单模型提升13.7个百分点。
- 适合需要高可靠性测试的CI/CD自动化场景。
人工软件测试成本高、耗时长,而单一智能体大模型易产生幻觉且行为不一致。本文提出一种多智能体委员会框架,多个具备视觉能力的大模型通过三轮投票机制协同决策,系统性探索网页应用。在84次实验中,9种测试角色、4种场景下,多智能体委员会整体任务成功率达89.5%。配置2至4个智能体时,成功率可达91.7%至100%,显著优于单智能体基线(78.0%),提升13.7至22.0个百分点。动作级成功率达93.1%,平均单步延迟0.71秒,支持实时连续集成测试。视觉智能体能准确识别界面元素,导航与报告成功率100%,表单填写成功率99.2%。在WebShop和OWASP基准测试中,分别取得74.7%和82.0%的成功率,覆盖8/10个OWASP Top 10漏洞类别。针对20个注入的回归缺陷,委员会检测F1得分为0.91,高于单智能体的0.78。开源实现支持可复现研究与实际部署。
原文摘要 · Abstract (English)
Manual software beta testing is costly and time-consuming, while single-agent large language model (LLM) approaches suffer from hallucinations and inconsistent behavior. We propose a multi-agent committee framework in which diverse vision-enabled LLMs collaborate through a three-round voting protocol to reach consensus on testing actions. The framework combines model diversity, persona-driven behavioral variation, and visual user interface understanding to systematically explore web applications. Across 84 experimental runs with 9 testing personas and 4 scenarios, multi-agent committees achieve an 89.5 percent overall task success rate. Configurations with 2 to 4 agents reach 91.7 to 100 percent success, compared to 78.0 percent for single-agent baselines, yielding improvements of 13.7 to 22.0 percentage points. At the action level, the system attains a 93.1 percent success rate with a median per-action latency of 0.71 seconds, enabling real-time and continuous integration testing. Vision-enabled agents successfully identify user interface elements, with navigation and reporting achieving 100 percent success and form filling achieving 99.2 percent success. We evaluate the framework on WebShop and OWASP benchmarks, achieving 74.7 percent success on WebShop compared to a 50.1 percent published GPT-3 baseline, and 82.0 percent success on OWASP Juice Shop security testing with coverage of 8 of the 10 OWASP Top 10 vulnerability categories. Across 20 injected regressions, the committee achieves an F1 score of 0.91 for bug detection, compared to 0.78 for single-agent baselines. The open-source implementation enables reproducible research and practical deployment of LLM-based software testing in CI/CD pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。