arXiv:2410.15185cs.RO2024-10被引 37

让机器人理解常识性安全规则,避免洒水、倾倒等语义风险。

Semantically Safe Robot Manipulation: From Semantic Scene Understanding to Motion Safeguards

  • 用大语言模型分析环境语义,识别不安全状态
  • 将语义不安全条件转化为可验证的安全动作约束
  • 在真实厨房中验证,兼顾碰撞与常识安全

在以人为中心的环境中确保机器人安全交互,需使其理解并遵守人类认知中的“常识”约束(如‘将盛水杯放在笔记本上方不安全,可能溢出’或‘旋转盛水杯不安全,易泼洒’)。尽管计算机视觉和机器学习已使机器人具备环境语义理解与推理能力,但现有安全决策研究很少融合语义理解。本文提出一种语义安全过滤框架,可对机器人输入进行语义约束(如不安全的空间关系、行为、姿态)与几何约束(如环境碰撞、自碰撞)的双重认证。给定感知输入后,构建3D环境语义地图,并利用大语言模型推断语义不安全条件;再通过控制屏障认证形式,将这些条件映射为安全动作。我们在远程操控抓取任务及真实厨房环境中应用学习到的扩散策略进行了验证,结果表明该方法能有效应对实际语义安全约束,实现超越传统避障的智能安全操作。

原文摘要 · Abstract (English)

Ensuring safe interactions in human-centric environments requires robots to understand and adhere to constraints recognized by humans as "common sense" (e.g., "moving a cup of water above a laptop is unsafe as the water may spill" or "rotating a cup of water is unsafe as it can lead to pouring its content"). Recent advances in computer vision and machine learning have enabled robots to acquire a semantic understanding of and reason about their operating environments. While extensive literature on safe robot decision-making exists, semantic understanding is rarely integrated into these formulations. In this work, we propose a semantic safety filter framework to certify robot inputs with respect to semantically defined constraints (e.g., unsafe spatial relationships, behaviors, and poses) and geometrically defined constraints (e.g., environment-collision and self-collision constraints). In our proposed approach, given perception inputs, we build a semantic map of the 3D environment and leverage the contextual reasoning capabilities of large language models to infer semantically unsafe conditions. These semantically unsafe conditions are then mapped to safe actions through a control barrier certification formulation. We demonstrate the proposed semantic safety filter in teleoperated manipulation tasks and with learned diffusion policies applied in a real-world kitchen environment that further showcases its effectiveness in addressing practical semantic safety constraints. Together, these experiments highlight our approach's capability to integrate semantics into safety certification, enabling safe robot operation beyond traditional collision avoidance.

机器人操作语义安全大模型常识推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。