首个评估大模型在真实入侵后响应能力的基准,发现现有模型难主动排查隐蔽攻击。
SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response

- 基于真实被攻陷云主机构建10个网络沙箱,模拟多种攻击路径与系统环境
- 23个前沿大模型在测试中均未实现任一沙箱的完整检测与修复,最高仅完成78%任务
- 适合安全研究者、红蓝对抗团队及大模型安全应用开发者参考
大型语言模型(LLM)代理正越来越多地应用于具有主机文件和命令行界面(CLI)访问权限的真实安全运营场景,因此全面评估其安全能力至关重要。然而,现有网络安全基准主要关注攻击前的纯净理想环境,对攻击后的响应场景研究不足。为此,我们提出了SecRespond,首个用于评估LLM代理在攻击后事件响应工作流中表现的基准。给定一个被攻陷主机的取证磁盘快照,以及主机安全产品报告的警报、漏洞扫描和基线检查结果,代理需生成关于入侵、基线风险和漏洞风险的取证报告,并制定修复方案。该任务在10个不同的网络沙箱中实现,每个沙箱源自一个独特的受攻陷云主机,涵盖4种入口类型、21种ATT&CK技术及5种操作系统。我们在OpenCode代理框架上评估了23个前沿大模型。实验结果显示,尽管当前代理能可靠识别警报所暴露的问题,但在主动排查磁盘以发现静默入侵,以及生成全面且可验证的修复计划方面表现不佳,没有任何模型在任一沙箱中实现完全检测与修复。这揭示了构建真实世界事件响应代理的根本瓶颈。该基准已公开于https://github.com/Alibaba-NLP/qqr/tree/main/data/secrespond。
原文摘要 · Abstract (English)
Large Language Model (LLM) agents are increasingly adopted in real-world security operations with access to host artifacts and command-line interfaces (CLIs), making it critical to thoroughly assess their security capabilities. However, existing cybersecurity benchmarks focus on pre-compromise settings where agents are placed in a clean and idealized environment before an attack occurs. This leaves the post-compromise setting underexplored. To address this gap, we introduce SecRespond, the first benchmark for evaluating LLM agents on the post-compromise incident-response workflow. Given a forensic disk snapshot of a compromised host together with the alerts, vulnerability scans, and baseline checks reported by a host security product, agents are required to produce forensic reports on intrusions, baseline risks, and vulnerability risks, together with a remediation plan. We instantiate this task across 10 cyber ranges, each constructed from a distinct compromised cloud host, spanning 4 entry-point types, 21 ATT&CK techniques, and 5 operating systems. We evaluate 23 frontier LLMs on the OpenCode agent harness. Experimental results show that although current agents can reliably uncover the problems exposed by alerts, they struggle to proactively investigate the disk for silent intrusions and to produce comprehensive, verified remediation plans, with no model achieving complete detection and remediation on any single range. This reveals a fundamental bottleneck in building agents for real-world incident response. The benchmark is publicly available at https://github.com/Alibaba-NLP/qqr/tree/main/data/secrespond.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。