让大模型用交互工具自主找漏洞，性能显著提升。
EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities
- 引入可运行调试器等交互工具的界面，支持动态交互。
- 在390个挑战中表现领先，多基准达顶尖水平。
- 发现模型自言自语现象，揭示新数据泄露机制。
尽管语言模型（LM）代理在编码和网页浏览等领域表现出色，但在网络安全领域的应用仍受限。本文提出EnIGMA，一个能自主解决夺旗（CTF）挑战的LM代理。通过引入新工具与交互界面，显著提升了代理对交互式终端程序中安全漏洞的发现与利用能力，首次实现大模型运行调试器、服务器连接工具等关键交互操作。在四个基准的390个挑战上进行实证分析表明，该方法在NYU CTF、Intercode-CTF和CyBench上均达到当前最优表现。最后，我们分析了数据泄露问题，提出新量化方法，并发现一种名为“自言自语”（soliloquizing）的新现象——模型在未与环境交互时自行生成虚假观察。代码与开发数据集已开源。
原文摘要 · Abstract (English)
Although language model (LM) agents have demonstrated increased performance in multiple domains, including coding and web-browsing, their success in cybersecurity has been limited. We present EnIGMA, an LM agent for autonomously solving Capture The Flag (CTF) challenges. We introduce new tools and interfaces to improve the agent's ability to find and exploit security vulnerabilities, focusing on interactive terminal programs. These novel Interactive Agent Tools enable LM agents, for the first time, to run interactive utilities, such as a debugger and a server connection tool, which are essential for solving these challenges. Empirical analysis on 390 CTF challenges across four benchmarks demonstrate that these new tools and interfaces substantially improve our agent's performance, achieving state-of-the-art results on NYU CTF, Intercode-CTF, and CyBench. Finally, we analyze data leakage, developing new methods to quantify it and identifying a new phenomenon we term soliloquizing, where the model self-generates hallucinated observations without interacting with the environment. Our code and development dataset are available at https://github.com/SWE-agent/SWE-agent/tree/v0.7 and https://github.com/NYU-LLM-CTF/NYU_CTF_Bench/tree/main/development respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。