测试生成新基准,让大模型主动发现代码漏洞。
TestExplora: Benchmarking LLMs for Proactive Bug Discovery via Repository-Level Test Generation
- 用文档作为标准,让大模型在真实仓库中主动找缺陷。
- 顶尖模型主动发现漏洞率仅16.06%,仍差距明显。
- 适合研究自动化测试与智能编程的开发者参考。
随着大语言模型(LLMs)在软件开发中应用日益广泛,软件保障需覆盖三项目标:防止回归、反应式复现和主动发现缺陷。现有评估普遍忽略第三项目标,或以现有代码为基准(合规陷阱),或依赖故障后数据(如问题报告),难以提前发现缺陷。为此,我们提出TestExplora,一个用于评估大模型在完整、真实仓库环境中作为主动测试者的基准。TestExplora包含482个仓库的2,389个任务,隐藏所有缺陷信号,模型需通过对比实现与文档推导的意图来主动发现漏洞,文档作为唯一判定依据。为确保评估可持续并减少信息泄露,我们提出持续性、时间感知的数据收集机制。评估结果显示显著能力差距:最先进模型最高仅达16.06%的失败转通过率(F2P)。进一步分析表明,处理跨模块复杂交互及采用代理式探索是关键。实验显示,以GPT-5-mini为基础的SWEAgent实现17.27% F2P与29.7% F2P@5,验证了代理探索在主动缺陷发现中的有效性与前景。
原文摘要 · Abstract (English)
Given that Large Language Models (LLMs) are increasingly applied to automate software development, comprehensive software assurance spans three distinct goals: regression prevention, reactive reproduction, and proactive discovery. Current evaluations systematically overlook the third goal. Specifically, they either treat existing code as ground truth (a compliance trap) for regression prevention, or depend on post-failure artifacts (e.g., issue reports) for bug reproduction-so they rarely surface defects before failures. To bridge this gap, we present TestExplora, a benchmark designed to evaluate LLMs as proactive testers within full-scale, realistic repository environments. TestExplora contains 2,389 tasks from 482 repositories and hides all defect-related signals. Models must proactively find bugs by comparing implementations against documentation-derived intent, using documentation as the oracle. Furthermore, to keep evaluation sustainable and reduce leakage, we propose continuous, time-aware data collection. Our evaluation reveals a significant capability gap: state-of-the-art models achieve a maximum Fail-to-Pass (F2P) rate of only 16.06%. Further analysis indicates that navigating complex cross-module interactions and leveraging agentic exploration are critical to advancing LLMs toward autonomous software quality assurance. Consistent with this, SWEAgent instantiated with GPT-5-mini achieves an F2P of 17.27% and an F2P@5 of 29.7%, highlighting the effectiveness and promise of agentic exploration in proactive bug discovery tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。