首个评估大模型在网络安全调查中表现的基准，基于真实日志构建可解释问题。
ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation
- 从微软云环境提取57类安全日志，构建威胁调查图并生成7542个问题。
- 最佳模型仅得0.606奖励分，任务难度高，仍有巨大提升空间。
- 适合研究大模型安全应用、自动化威胁分析的团队使用。
我们提出ExCyTIn-Bench，首个通过源自调查图的安全问题评估大模型代理（LLM agent）在网络安全调查任务中的表现的基准。真实安全分析师需处理大量异构安全日志，并追踪多跳证据链来调查威胁。随着大模型的发展，构建基于大模型的自动威胁调查代理成为有前景的方向。我们从受控的Azure租户中构建基准，包含来自Microsoft Sentinel及其相关服务的57张日志表，生成了7542个问题。利用专家设计的检测逻辑提取安全日志，构建威胁调查图，再通过大模型以图上配对节点生成问题：起始节点作为背景上下文，结束节点作为答案。将每个问题锚定于明确的节点和边，不仅提供自动且可解释的真值答案，还使流程可复用、易于扩展至新日志。在测试集上的综合实验表明该任务极具挑战性：当前最佳模型仅达0.606奖励分数，为未来研究留下广阔空间。代码已开源：https://github.com/microsoft/SecRL。
原文摘要 · Abstract (English)
We present ExCyTIn-Bench, the first benchmark to Evaluate an LLM agent X on the task of Cyber Threat Investigation through security questions derived from investigation graphs. Real-world security analysts must sift through a large number of heterogeneous security logs, follow multi-hop chains of evidence to investigate threats. With the developments of LLMs, building LLM-based agents for automatic threat investigation is a promising direction. We construct a benchmark from a controlled Azure tenant including a SQL environment covering 57 log tables from Microsoft Sentinel and related services, and 7542 generated questions. We leverage security logs extracted with expert-crafted detection logic to build threat investigation graphs, and then generate questions with LLMs using paired nodes on the graph, taking the start node as background context and the end node as answer. Anchoring each question to these explicit nodes and edges not only provides automatic, explainable ground truth answers but also makes the pipeline reusable and readily extensible to new logs. Our comprehensive experiments on the test set with different models confirm the difficulty of the task: the best model so far can achieve a reward of 0.606, leaving much headroom for future research. The code is available at https://github.com/microsoft/SecRL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。