构建多目标网页漏洞竞赛基准，评估大模型在未知环境中的策略探索能力。
CTFExplorer: Evaluating LLM Offensive Agents Through Multi-Target Web CTF Benchmarking
- 设计40个集成的网页漏洞服务，模拟真实攻防中多目标探索场景。
- 支持自主发现、优先级排序与攻击链构建，首次实现对策略思维的量化评估。
- 适合研究智能攻防、安全自动化及大模型推理能力的研究者。
现有基于大语言模型的进攻型安全代理评测采用孤立的单目标设置，仅针对已知漏洞服务和固定目标进行评估，虽能有效衡量攻击成功率，却忽略了真实攻防比赛中参与者对未知系统的排查、目标优先级判断与资源分配的策略性决策。当前评估体系难以衡量超越单一攻击之外的战略推理能力。为此，我们提出 extit{CTFExplorer}，一个面向多目标的网页类攻防竞赛基准，推动评估从单一攻击转向探索、优先级判断与攻击链组合能力的测试。该基准在一个环境中部署40个网页漏洞服务，要求代理在无预设指引下自主发现、区分并利用目标。我们还引入一种反应式多代理架构作为基线框架，并开发了不依赖具体代理的评估机制，可记录结构化推理轨迹，实现细粒度行为分析。该框架不仅关注是否成功获取旗帜，还能评估代理的目标选择、失败假设处理、多阶段协作与安全情报提取等能力。
原文摘要 · Abstract (English)
Existing benchmarks for LLM-based offensive security agents use isolated, single-target setups with a known vulnerable service and fixed objective. They measure exploitation effectively, but miss how real Capture-the-Flag (CTF) participants triage unknown surfaces, prioritize targets, and allocate effort under uncertainty. Current evaluations therefore fail to assess strategic reasoning beyond exploitation alone. To address this, we introduce \textit{CTFExplorer}, a benchmark suite that shifts offensive security evaluation toward a multi-target setting, which tests how agents explore, prioritize, and chain attacks. CTFExplorer deploys 40 web-based vulnerable services within a single environment, where agents must autonomously discover, distinguish, and exploit targets without predefined guidance. We also present a reactive multi-agent setup as a reference agent framework and develop an agent-agnostic evaluation framework that records structured reasoning traces for fine-grained assessment. This enables behavioral evaluation beyond binary flag capture, such as how agents manage target selection, handle failed hypotheses, coordinate across multiple stages, and extract security intelligence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。