测试大模型生成漏洞利用代码的能力，发现GPT-4o最配合但仍未成功。
Good News for Script Kiddies? Evaluating Large Language Models for Automated Exploit Generation
- 设计真实安全实验环境，用重构的软件实验室测试大模型
- GPT-4o生成错误最少，但所有模型均未成功生成有效漏洞利用
- 适合关注大模型安全风险的研究者和安全防御开发者
大型语言模型（LLMs）在代码任务中表现突出，引发其被用于自动化漏洞利用（AEG）的担忧。本文首次系统评估了大模型在AEG中的有效性，考察其合作意愿与技术能力。为减少数据集偏差，我们引入一个包含五个重构版软件安全实验的基准测试。同时，设计基于LLM的攻击者，系统化地向模型发起漏洞利用生成请求。实验表明，GPT-4和GPT-4o表现出高合作性，接近无限制模型；而Llama3最为抗拒。然而，所有模型均未能成功生成针对重构实验环境的有效漏洞利用代码，尽管GPT-4o产生的错误最少，显示出大模型驱动漏洞利用发展的潜在可能。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable capabilities in code-related tasks, raising concerns about their potential for automated exploit generation (AEG). This paper presents the first systematic study on LLMs' effectiveness in AEG, evaluating both their cooperativeness and technical proficiency. To mitigate dataset bias, we introduce a benchmark with refactored versions of five software security labs. Additionally, we design an LLM-based attacker to systematically prompt LLMs for exploit generation. Our experiments reveal that GPT-4 and GPT-4o exhibit high cooperativeness, comparable to uncensored models, while Llama3 is the most resistant. However, no model successfully generates exploits for refactored labs, though GPT-4o's minimal errors highlight the potential for LLM-driven AEG advancements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。