测试大模型代理在技能接口下的安全漏洞，发现非用户攻击也能引发危险行为。
SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces
- 构建可运行的基准测试，模拟技能层面的安全攻击
- 155个攻击案例揭示不同领域失败模式各异
- 适合关注智能体安全与可信执行环境的研究者
可复用技能正成为扩展大语言模型代理的常见接口，将程序指导与文件、工具、记忆及执行环境关联。然而，这种模块化引入了现有安全评估未覆盖的攻击面：即使用户请求无害，技能引导、本地资源或执行环境文件仍可能诱导代理产生不安全行为。本文提出SkillSafetyBench，一个可运行的基准测试，用于评估此类面向技能的安全失效。该基准涵盖47个任务、6个风险领域、30个安全类别，共155个对抗性案例，每例均通过特定规则验证器评估。多组命令行代理与模型后端实验表明，非用户攻击能持续引发不安全行为，且失败模式在领域、攻击方法和模型组合间存在显著差异。结果表明，代理安全不仅依赖模型对齐，更取决于其对技能的理解、对工作流上下文的信任程度以及在可执行环境中的行动方式。完整基准已开源：https://github.com/AI45Lab/skill-safety-bench。
原文摘要 · Abstract (English)
Reusable skills are becoming a common interface for extending large language model agents, packaging procedural guidance with access to files, tools, memory, and execution environments. However, this modularity introduces attack surfaces that are largely missed by existing safety evaluations: even when the user request is benign, unsafe influence may reside in skill guidance, local artifacts, or execution-environment files that steer the agent toward unsafe actions. We present SkillSafetyBench, a runnable benchmark for evaluating such skill-facing safety failures. SkillSafetyBench includes 155 adversarial cases across 47 tasks, 6 risk domains, and 30 safety categories, each evaluated with a case-specific rule-based verifier. Experiments with multiple CLI agents and model backends show that non-user attacks can consistently induce unsafe behavior, with distinct failure patterns across domains, attack methods, and scaffold-model pairings. Our findings suggest that agent safety depends not only on model-level alignment, but also on how agents interpret skills, trust workflow context, and act through executable environments. The complete benchmark is available at https://github.com/AI45Lab/skill-safety-bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。