用形式化逻辑层层验证大模型智能体的物理安全,防患于未然。
SENTINEL: A Multi-Level Formal Framework for Safety Evaluation of Foundation Model-based Embodied Agents
- 将安全要求转为时序逻辑公式,从理解、规划到执行多层验证。
- 在VirtualHome和AI2-THOR中验证多个智能体,发现潜在安全漏洞。
- 适合研究智能体安全、具身AI验证的开发者与研究人员。
我们提出SENTINEL,一个用于评估基础模型(FM)驱动的具身智能体物理安全性的多层次形式化框架。SENTINEL是首个在统一形式框架下,对语义理解、计划生成和物理执行三个层级进行安全评估的方法。不同于依赖启发式规则或主观判断的现有方法,SENTINEL将实际安全需求基于形式时序逻辑(TL)语义,精确描述状态不变量、时间依赖关系和时序约束。该框架采用多级验证流程:(i) 语义层,将自然语言安全要求形式化为TL公式,并检验智能体理解是否与公式对齐;(ii) 计划层,验证智能体生成的高层动作计划与子目标是否违反TL公式,提前发现不安全计划;(iii) 轨迹层,将多个执行轨迹合并为计算树,高效验证其是否满足详尽的物理级TL规范。我们在VirtualHome和AI2-THOR环境中应用SENTINEL,对多个基于大模型的具身智能体进行多样化安全要求的形式化评估。实验表明,通过在多层级上结合时序逻辑与验证方法,SENTINEL为仿真物理环境中大模型智能体的安全性评估提供了严谨基础,能有效暴露任务理解、规划与执行阶段的潜在安全违规。
原文摘要 · Abstract (English)
We present SENTINEL, a framework for formally evaluating the physical safety of foundation model (FM)-based embodied agents. SENTINEL is the first to provide multi-level safety evaluation across semantic interpretation, plan generation, and physical execution within a unified formal framework. Unlike prior methods that rely on heuristic rules or subjective FM judgments, SENTINEL grounds practical safety requirements in formal temporal logic (TL) semantics that can precisely specify state invariants, temporal dependencies, and timing constraints. It employs a multi-level verification pipeline where (i) at the semantic level, intuitive natural language safety requirements are formalized into TL formulas and the agent's understanding of these requirements is probed for alignment with the TL formulas; (ii) at the plan level, high-level action plans and subgoals generated by the agent are verified against the TL formulas to detect unsafe plans before execution; and (iii) at the trajectory level, multiple execution trajectories are merged into a computation tree and efficiently verified against physically-detailed TL specifications for a final safety check. We apply SENTINEL in VirtualHome and AI2-THOR, and formally evaluate multiple FM-based embodied agents against diverse safety requirements. Our experiments show that by grounding physical safety in temporal logic and applying verification methods across multiple levels, SENTINEL provides a rigorous foundation for systematically evaluating the safety of FM-based embodied agents in simulation-based physical environments, and can effectively expose potential safety violations in interpreting, planning, and executing the tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。