用代码变换生成可比安全挑战，评估大模型在变体下的鲁棒性。
Capture the Flags: Family-Based Evaluation of Agentic LLMs via Semantics-Preserving Transformations
- 通过语义保持变换生成同一题目的多个变体，固定攻击策略。
- 模型对重命名和插入代码鲁棒，但复杂混淆使成功率下降23%。
- 适合研究模型安全推理能力或评估工具调用性能的开发者。
代理型大语言模型（LLM）在网络安全任务中越来越多地使用攻防赛（CTF）基准进行评估，但现有单点式基准难以揭示模型在源代码不同变体下的鲁棒性与泛化能力。本文提出CTF挑战族，通过语义保持的程序变换从单一CTF生成一组语义等价的挑战，从而在固定漏洞利用策略的前提下，实现对模型鲁棒性的可控评估。我们构建了Evolve-CTF工具，可基于Python挑战应用多种变换生成挑战族。利用该工具从Cybench和Intercode挑战中生成系列挑战，评估了13种带工具访问权限的代理型LLM配置。结果表明：模型对变量重命名和代码插入具有高度鲁棒性；而复合变换和深度混淆会使性能显著下降，需更复杂的工具使用策略。显式推理对成功率影响微弱。本工作贡献了一种新型评估方法与工具，并提供了大规模数据集，用于刻画当前顶尖模型在此领域的实际能力。
原文摘要 · Abstract (English)
Agentic large language models (LLMs) are increasingly evaluated on cybersecurity tasks using capture-the-flag (CTF) benchmarks, yet existing pointwise benchmarks offer limited insight into agent robustness and generalisation across alternative versions of the source code. We introduce CTF challenge families, whereby a single CTF is used to generate a family of semantically-equivalent challenges via semantics-preserving program transformations, enabling controlled evaluation of robustness while keeping the underlying exploit strategy fixed. We present Evolve-CTF, a tool that generates CTF families from Python challenges using a range of transformations. Using Evolve-CTF to derive families from Cybench and Intercode challenges, we evaluate 13 agentic LLM configurations with tool access. We find that models are remarkably robust to renaming and code insertion, but that composed transformations and deeper obfuscation degrade performance by requiring more sophisticated tool use. Enabling explicit reasoning has little effect on success rates. Our work contributes a technique and tool for future LLM evaluations, and a large dataset characterising the capabilities of current state-of-the-art models in this domain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。