首个评估具身大模型安全规划能力的基准,覆盖真实场景中的显性和隐性风险。
SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents

- 构建交互式仿真环境与750个高质任务,涵盖10类危险和3种任务类型
- 多模型测试显示安全意识普遍薄弱,最优方案仅10%拒绝高危任务
- 适合关注AI安全、具身智能部署的研究者与开发者使用
随着大语言模型(LLMs)的融入,具身智能体具备理解并规划复杂自然语言指令的能力。然而,这些智能体也可能完美执行某些危险任务,对现实世界造成损害。现有基准大多忽视关键安全风险,仅关注规划性能;少数评估仅基于非交互的图文数据。为此,我们提出SafeAgentBench——首个面向具身LLM智能体在交互式仿真环境中安全意识任务规划的综合性基准,涵盖显性和隐性危害。该基准包含:(1) 可执行、多样且高质量的750项任务数据集,经严格筛选覆盖10类潜在风险与3种任务类型;(2) SafeAgentEnv,一个通用具身环境,配备底层控制器,支持多智能体执行,提供17种高层动作,适配9个前沿基线模型;(3) 从执行与语义双视角出发的可靠评估方法。实验结果表明,尽管不同设计框架的智能体在任务成功率上差异显著,其整体安全意识仍较弱,最安全基线对详细高危任务的拒绝率仅为10%。仅更换驱动智能体的LLM,并未带来显著的安全意识提升。数据集与代码已开源于https://github.com/shengyin1224/SafeAgentBench和https://huggingface.co/datasets/safeagentbench/SafeAgentBench。
原文摘要 · Abstract (English)
With the integration of large language models (LLMs), embodied agents have strong capabilities to understand and plan complicated natural language instructions. However, a foreseeable issue is that those embodied agents can also flawlessly execute some hazardous tasks, potentially causing damages in the real world. Existing benchmarks predominantly overlook critical safety risks, focusing solely on planning performance, while a few evaluate LLMs' safety awareness only on non-interactive image-text data. To address this gap, we present SafeAgentBench -- the first comprehensive benchmark for safety-aware task planning of embodied LLM agents in interactive simulation environments, covering both explicit and implicit hazards. SafeAgentBench includes: (1) an executable, diverse, and high-quality dataset of 750 tasks, rigorously curated to cover 10 potential hazards and 3 task types; (2) SafeAgentEnv, a universal embodied environment with a low-level controller, supporting multi-agent execution with 17 high-level actions for 9 state-of-the-art baselines; and (3) reliable evaluation methods from both execution and semantic perspectives. Experimental results show that, although agents based on different design frameworks exhibit substantial differences in task success rates, their overall safety awareness remains weak. The most safety-conscious baseline achieves only a 10% rejection rate for detailed hazardous tasks. Moreover, simply replacing the LLM driving the agent does not lead to notable improvements in safety awareness. Dataset and codes are available in https://github.com/shengyin1224/SafeAgentBench and https://huggingface.co/datasets/safeagentbench/SafeAgentBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。