构建CTF知识测评基准，提升大模型实战解题能力
Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challenges
- 设计专注的CTFKnow评测集，精准衡量模型技术知识
- 提出CTFAgent框架，使模型在两大数据集上性能提升超80%
- 适合关注安全攻防与AI自动化研究的读者
Capture-the-Flag（CTF）竞赛对网络安全教育至关重要。随着大语言模型（LLMs）的发展，其自动化解决CTF挑战的能力日益受到关注。例如，DARPA自2023年起举办AIxCC竞赛，推动基于AI的自动攻防技术发展。然而，这需要综合知识、推理和行动能力。本文强调技术知识在解决CTF问题中的核心作用，专门构建了一个包含3,992个问题的聚焦型评测基准CTFKnow，以衡量LLMs在该方面的表现。研究发现，尽管LLMs具备丰富技术知识，但在具体场景中准确应用及根据环境反馈调整策略方面仍存在不足。基于此，我们提出CTFAgent框架，引入两阶段检索增强生成（RAG）和交互式环境增强模块，分别提升模型的技术知识掌握与漏洞利用能力。实验显示，该框架在两个主流CTF数据集上均实现超过80%的性能提升；在卡内基梅隆大学主办的picoCTF2024竞赛中，排名接近7,000支参赛队伍的前23.6%。结果验证了测评研究的价值与框架在提升模型实战能力上的潜力。
原文摘要 · Abstract (English)
Capture-the-Flag (CTF) competitions are crucial for cybersecurity education and training. As large language models (LLMs) evolve, there is increasing interest in their ability to automate CTF challenge solving. For example, DARPA has organized the AIxCC competition since 2023 to advance AI-powered automated offense and defense. However, this demands a combination of multiple abilities, from knowledge to reasoning and further to actions. In this paper, we highlight the importance of technical knowledge in solving CTF problems and deliberately construct a focused benchmark, CTFKnow, with 3,992 questions to measure LLMs' performance in this core aspect. Our study offers a focused and innovative measurement of LLMs' capability in understanding CTF knowledge and applying it to solve CTF challenges. Our key findings reveal that while LLMs possess substantial technical knowledge, they falter in accurately applying this knowledge to specific scenarios and adapting their strategies based on feedback from the CTF environment. Based on insights derived from this measurement study, we propose CTFAgent, a novel LLM-driven framework for advancing CTF problem-solving. CTFAgent introduces two new modules: two-stage Retrieval Augmented Generation (RAG) and interactive Environmental Augmentation, which enhance LLMs' technical knowledge and vulnerability exploitation on CTF, respectively. Our experimental results show that, on two popular CTF datasets, CTFAgent both achieves over 80% performance improvement. Moreover, in the recent picoCTF2024 hosted by CMU, CTFAgent ranked in the top 23.6% of nearly 7,000 participating teams. This reflects the benefit of our measurement study and the potential of our framework in advancing LLMs' capabilities in CTF problem-solving.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。