对比基础代码代理与复杂架构,发现简单模型也能解决多数渗透测试任务。
Baselines Before Architecture: Evaluating Coding Agents for Autonomous Penetration Testing
- 用相同模型和规则测试基础代码代理,作为性能基准
- 新模型升级使同一框架下任务覆盖率显著提升
- 建议评估时先报告基础代理表现,再归因于架构改进
近期自主渗透测试研究在前沿大模型基础上添加多组件安全封装,宣称取得高分。但因同时更换模型与架构,难以判断性能提升来自何处。本文在104项任务的XBOW基准上,采用默认代码命令行代理作为基础基准,统一使用GPT-5模型、预算、目标接口和评分规则,测试Codex、OpenCode和Pi的表现。结果识别出最强同模型基线,并检验安全提示变体是否提升分数。随后将默认Codex框架与公开的MAPTA及PentestGPT V2在最接近的模型匹配下对比。最后在GPT-5.2和GPT-5.5上重复实验,验证模型规模对同一框架的影响。结果显示:专用封装可带来可测量的分数提升,可能提高成本效率;但基础代码代理已解决大量任务;重复运行基础代理可达到甚至超过部分已有架构的联合覆盖率;新模型显著提升同一框架表现。未来评估应先报告模型匹配的基础代理结果,避免仅归因于架构设计。
原文摘要 · Abstract (English)
Recent autonomous penetration testing papers report high benchmark scores while adding multi-component security harnesses around frontier LLMs. Because these systems often change both architecture and backbone model, it is difficult to tell how much performance comes from the harness rather than from the underlying model. This paper presents a controlled study on the 104-task XBOW benchmark using default coding CLI agents as plain-agent baselines. We first run Codex, OpenCode, and Pi with the same GPT-5 model, budget, target interface, and scoring rule. This phase identifies the strongest same-model baseline and tests whether security-specific prompt variants improve its observed score. We then compare the default Codex scaffold with published MAPTA and PentestGPT V2 results under the closest available model matches. Finally, we repeat the plain-agent experiment with GPT-5.2 and GPT-5.5 to measure model scaling inside the same scaffold. The results show a mixed but practical picture. Specialised harnesses can add measurable benchmark lift and may improve cost efficiency, but plain coding agents already solve a large share of the benchmark; repeated plain-agent runs can match or exceed some published architecture scores in union coverage, and newer models substantially improve the same scaffold. Future evaluations should report model-matched plain-agent baselines before attributing benchmark gains to architecture design alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。