测试大模型在真实攻击日志中自主找恶意时间戳的能力，发现表现极差。
Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps
- 构建游戏化环境模拟真实攻防日志，让模型自主查可疑时间点。
- 顶级模型平均只识别3.8%的恶意事件，无一模型能全数找出。
- 结果揭示大模型不适应开放式威胁狩猎，适合安全运维者警惕使用。
我们提出网络安全防御基准（Cyber Defense Benchmark），用于评估大语言模型（LLM）代理在安全运营中心（SOC）核心任务——威胁狩猎中的表现：给定一个未经提示的原始Windows事件日志数据库，需准确找出恶意事件的时间戳。该基准将来自OTRF Security-Datasets语料库的106个真实攻击流程（涵盖12种战术、86个MITRE ATT&CK子技术）封装进Gymnasium强化学习环境。每个实验周期提供一个内存中含7.5万至13.5万条记录的SQLite数据库，由确定性攻防模拟器生成，包含时间偏移和实体混淆。模型需通过迭代提交SQL查询发现并明确标记恶意时间戳，评分方式为类似黑客夺旗赛（CTF）的Sigma规则真值比对。在26个攻击场景（覆盖105/106个流程）上评估五个前沿模型（Claude Opus 4.6、GPT-5、Gemini 3.1 Pro、Kimi K2.5、Gemini 3 Flash），发现所有模型表现严重不足：最佳模型（Claude Opus 4.6）平均仅正确标记3.8%的恶意事件，且没有任何一次运行能完全发现所有恶意时间戳。我们定义及格标准为每项ATT&CK战术≥50%召回率——这是无监督部署的最低门槛。结果表明：无一模型达标，领先者仅在5/13个战术上达到该标准，其余均未通过。这说明当前大模型虽在结构化问答中表现良好，却仍不适用于开放式、证据驱动的威胁狩猎。
原文摘要 · Abstract (English)
We introduce the Cyber Defense Benchmark, a benchmark for measuring how well large language model (LLM) agents perform the core SOC analyst task of threat hunting: given a database of raw Windows event logs with no guided questions or hints, identify the exact timestamps of malicious events. The benchmark wraps 106 real attack procedures from the OTRF Security-Datasets corpus - spanning 86 MITRE ATT&CK sub-techniques across 12 tactics - into a Gymnasium reinforcement-learning environment. Each episode presents the agent with an in-memory SQLite database of 75,000-135,000 log records produced by a deterministic campaign simulator that time-shifts and entity-obfuscates the raw recordings. The agent must iteratively submit SQL queries to discover malicious event timestamps and explicitly flag them, scored CTF-style against Sigma-rule-derived ground truth. Evaluating five frontier models - Claude Opus 4.6, GPT-5, Gemini 3.1 Pro, Kimi K2.5, and Gemini 3 Flash - on 26 campaigns covering 105 of 106 procedures, we find that all models fail dramatically: the best model (Claude Opus 4.6) submits correct flags for only 3.8% of malicious events on average, and no run across any model ever finds all flags. We define a passing score as >= 50% recall on every ATT&CK tactic - the minimum bar for unsupervised SOC deployment. No model passes: the leader clears this bar on 5 of 13 tactics and the remaining four on zero. These results suggest that current LLMs are poorly suited for open-ended, evidence-driven threat hunting despite strong performance on curated Q&A security benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。