arXiv:2606.02302cs.CRcs.AI2026-06

用规范生成安全任务,评估自主智能体的潜在风险行为。

SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents

论文配图:SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents
图 1 · 摘自论文原文
  • 根据安全规范自动生成多样化攻击任务
  • 覆盖资源、用户任务、环境等多类风险场景
  • 支持对执行过程的细粒度安全评估,适合研究者使用

自主大模型智能体在状态化环境中调用工具、文件、内存和外部服务,虽能完成复杂任务,但也带来难以捕捉的安全风险。现有评测基准多依赖人工设计任务,覆盖范围有限,且仅关注最终结果而非执行过程中的不安全行为。我们提出 SeClaw 框架,结合规范驱动的安全任务生成与基于执行过程的评估方法。通过结构化风险规范,可规模化、可控地生成安全任务;SeClaw Docker 提供标准化测试环境,支持对资源、用户任务、环境及智能体自身行为引发的风险进行评估,并实现对执行轨迹中不安全操作的追踪。该框架为衡量、诊断和比较自主智能体的安全缺陷提供了可复现的基础。代码已开源:https://github.com/seclaw-eval/seclaw-eval。

原文摘要 · Abstract (English)

Autonomous LLM agents increasingly operate in stateful environments where they access tools, files, memory, and external services. While such capabilities enable complex real-world workflows, they also introduce security risks that are difficult to capture with existing evaluations. Current agent security benchmarks often rely on manually curated tasks, provide limited coverage of emerging threats, and focus primarily on final outcomes rather than the execution processes that lead to unsafe behavior. We introduce SeClaw, a framework that combines specification-driven security task synthesis with execution-based security evaluation for Autonomous agents. Spec-driven security task synthesis enables scalable and controllable construction of security tasks from structured risk specifications, while SeClaw docker provides a standardized testbed for evaluating agent behavior under diverse safety-risk scenarios. The benchmark covers risks arising from resources, user tasks, environments, and intrinsic agent behaviors, and supports trajectory-aware assessment of unsafe actions beyond final responses. By bridging systematic task synthesis and reproducible security evaluation, SeClaw provides a practical foundation for measuring, diagnosing, and comparing security failures in autonomous LLM agents. The code is available at https://github.com/seclaw-eval/seclaw-eval.

智能体安全安全评测LLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。