构建安全检测基准,验证开放智能体技能的恶意行为。
Benchmarking Security Risk Detection and Verification in Open Agentic Skill Ecosystems

- 分两阶段检测:先语义分析说明书,再沙箱执行看运行行为。
- 89%恶意技能被传统方法漏检,高权限操作是主要攻击入口。
- 提供真实攻击案例的运行证据,适合安全研究人员参考。
开放智能体平台允许社区成员发布可复用的技能,供智能体在运行时调用。这种可扩展性也带来了供应链风险:恶意贡献者可在看似无害的技能中隐藏有害行为。然而,现有防御措施难以评估,因缺乏同时衡量恶意技能检测与运行时验证的基准。我们提出SkillVetBench,一个针对开放智能体技能生态系统的双阶段安全审核基准。第一阶段对每个技能的自然语言说明进行语义审核,以发现隐藏的恶意意图;第二阶段在受监控的沙箱中执行被标记的技能,观察运行时行为并收集可审计的证据。基准基于真实开放生态系统OpenClaw中的已确认恶意技能,包含近期ClawHavoc供应链攻击的样本。与仅依赖静态分析的方法不同,SkillVetBench通过执行轨迹验证检测到的威胁。实验表明:(1) 仅依赖语义或签名的基线方法不足,会漏掉高达89%的恶意技能,这些威胁源于自然语言指令、多组件逻辑或跨组件交互;(2) 运行时攻击集中在少数高权限原语,尤其是exec、write_file、install_skill和spawn;(3) SkillVetBench提供了案例研究,证明沙箱执行能直接提供支持恶意判定的明确运行证据。
原文摘要 · Abstract (English)
Open agent platforms allow community contributors to publish reusable skills that agents can invoke at runtime. This extensibility also creates a supply-chain risk: malicious contributors can hide harmful behavior inside skills that appear benign under superficial inspection. However, existing defenses are hard to evaluate because there is no benchmark that measures both malicious-skill detection and runtime verification. We present SkillVetBench, a two-stage security vetting benchmark for open agentic skill ecosystems. The first stage performs semantic vetting over each skill's natural-language specification to detect hidden malicious intent. The second stage executes flagged skills in an instrumented sandbox to observe runtime behavior and collect auditable evidence. We build a benchmark from confirmed malicious skills in the live OpenClaw ecosystem, including samples from the recent ClawHavoc supplychain campaign. Unlike static-only methods, SkillVetBench verifies detected threats with execution traces. Our experiments show that: (1) semantic-only and signature-based baselines are insufficient, missing up to 89\% of malicious skills whose threats arise from natural-language instructions, multicomponent logic, or cross-component interactions; (2) runtime attacks are concentrated in a small set of high-permission primitives, especially exec, write\_file, install\_skill, and spawn; and (3) SkillVetBench provides case studies in which sandbox execution directly supports malicious verdicts with concrete runtime evidence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。