用大模型实现自动渗透测试，可自主生成指令并评估攻防效果。
HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing
- 构建双模块代理，分阶段规划命令与总结反馈。
- 在200个挑战上测试，GPT-4o表现优于官方预期。
- 开源框架支持安全可控的自动化攻防研究。
我们提出HackSynth，一种基于大语言模型（LLM）的自主渗透测试代理。其双模块架构包含规划器（Planner）和摘要器（Summarizer），可迭代生成命令并处理反馈。为评估该代理，我们设计了两个基于CTF的新基准集，分别使用PicoCTF和OverTheWire平台，涵盖200个不同领域与难度的挑战，提供标准化评测框架。实验分析了HackSynth的核心参数，包括创造力（temperature和top-p）和令牌利用率。测试覆盖多个开源与专有LLM。结果表明，使用GPT-4o时性能最佳，超出其系统卡描述。同时讨论了行动的安全性与可预测性。研究显示LLM代理在推动自主渗透测试方面潜力巨大，且需强化防护机制。HackSynth及基准数据集已开源，旨在促进自主网络安全研究。
原文摘要 · Abstract (English)
We introduce HackSynth, a novel Large Language Model (LLM)-based agent capable of autonomous penetration testing. HackSynth's dual-module architecture includes a Planner and a Summarizer, which enable it to generate commands and process feedback iteratively. To benchmark HackSynth, we propose two new Capture The Flag (CTF)-based benchmark sets utilizing the popular platforms PicoCTF and OverTheWire. These benchmarks include two hundred challenges across diverse domains and difficulties, providing a standardized framework for evaluating LLM-based penetration testing agents. Based on these benchmarks, extensive experiments are presented, analyzing the core parameters of HackSynth, including creativity (temperature and top-p) and token utilization. Multiple open source and proprietary LLMs were used to measure the agent's capabilities. The experiments show that the agent performed best with the GPT-4o model, better than what the GPT-4o's system card suggests. We also discuss the safety and predictability of HackSynth's actions. Our findings indicate the potential of LLM-based agents in advancing autonomous penetration testing and the importance of robust safeguards. HackSynth and the benchmarks are publicly available to foster research on autonomous cybersecurity solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。