用大模型当裁判，全方位评估开源智能体技能的安全风险
SkillVetBench: LLM-as-Judge for Multi-Dimensional Security Risk Evaluation in Open-Source LLM Agent Skills
- 构建多维评分体系SARS，从指令执行等五个维度评估风险
- 对78个已知恶意技能实现零漏报，22个正常技能零误报
- 适合安全研究人员、开源项目维护者和大模型应用开发者
开源大模型智能体生态快速发展，但社区贡献的技能（扩展能力的模块化工具）安全性仍缺乏有效评估。现有扫描器仅在代码层工作，无法识别指令层与多智能体风险——如通过自然语言指令劫持智能体、利用编码侧信道泄露数据或串联攻击链。为此，我们提出SKILLVETBENCH，一个基于LLM-as-Judge的实时公开排行榜，集成五维风险评分体系SARS、CVSS v4.0向量分解及ClawHub双视角对比（大模型评审与官方市场判罚）。基于配套基准论文[1]，该系统在78个确认恶意技能上实现零漏报，在22个良性控制样本上零误报；最佳静态基线SKILLSIEVE仍遗漏15%威胁；对提示注入、内存污染等指令层风险，传统工具漏检率达89%至100%（如CODEBERT未检测出9个内存污染技能）。四个LLM评估器检测率在35%至95%之间波动，表明生产环境需采用集成评分策略。
原文摘要 · Abstract (English)
Open-source LLM agent ecosystems are growing rapidly, yet the security of community-contributed skills - modular tool definitions that extend agent capabilities - remains largely unvetted. The gap we fill: existing scanners operate at the code layer and are structurally blind to instruction-layer and multi-agent risk - natural-language directives that hijack an agent, exfiltrate data through encoded side channels, or chain harm across pipelines - so what is needed is a semantic, multi-dimensional vetting system rather than another signature matcher. We present SKILLVETBENCH, a live public leaderboard on Hugging Face that uses an LLM-as-Judge to vet agent skills. What is new: SARS (Skill Agentic Risk Score), a five-dimensional agentic-risk metric with a principled weighted formula for instruction-following systems. What is integrated: full CVSS v4.0 vector decomposition and a ClawHub dual-view that places our LLM-generated review beside the official marketplace verdict. What is demonstrated: drawing on our companion benchmark paper [ 1], the LLM-as-Judge stage achieves zero false negatives across 78 confirmed-malicious skills and zero false positives across 22 benign controls, while the best static baseline (SKILLSIEVE) still misses 15%; for instruction-layer categories such as Prompt Injection and Memory Poisoning, conventional tools miss between 89% and 100% of threats (e.g., CODEBERT detects none of nine memory-poisoning skills). Detection rates vary from 35% to 95% across four LLM evaluators, motivating ensemble scoring in production deployments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。