新基准揭示视觉语言动作模型成功但不安全的隐忧
SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models
- 用信号时序逻辑定义任务相关安全规则，量化执行过程中的安全隐患
- 发现高成功率模型仍有13%-15%的不安全轨迹，厨房任务中超一半成功样本违反安全条款
- 适用于评估机器人操作安全性，尤其关注真实场景中隐性风险
视觉-语言-动作（VLA）基准通常只衡量任务是否完成，却忽略了执行过程中的安全问题：如过度接触、扰动周围物体、持物不稳或自身碰撞。我们提出SafeVLA-Bench，一个后置式安全评估框架，将任务相关的安全要求形式化为信号时序逻辑（STL）规范，并报告原生成功率及两个不安全成功指标：Succ-But-Unsafe（SBU），即既成功又违反安全的轨迹比例；以及违规严重度指数（VSI），反映最严重违规程度。我们在LIBERO和RoboCasa-365上实例化该框架，评估了九个策略在桌面与厨房操作任务中的表现。结果显示，高任务成功率并不等于安全执行：高成功率的桌面基线仍存在13%至15%的不安全轨迹率，而RoboCasa-365中36%至56%的成功轨迹至少违反一项活跃安全条款。
原文摘要 · Abstract (English)
Vision-language-action (VLA) benchmarks measure whether a policy completes a requested manipulation task, but binary success can hide safety-relevant trajectory behavior: reaching the goal while applying excessive contact, disturbing bystander objects, destabilizing the held object, or entering robot self-contact. We present SafeVLA-Bench, a post-hoc safety-evaluation framework for existing simulator-based VLA benchmarks. It formalizes task-aware safety requirements as Signal Temporal Logic (STL) specifications and reports native success with two unsafe-success metrics: Succ-But-Unsafe (SBU), the fraction of rollouts that both succeed and violate safety, and Violation Severity Index (VSI), a bounded worst-violation depth score. We instantiate SafeVLA-Bench on LIBERO and RoboCasa-365, evaluating nine policy-benchmark entries across tabletop and kitchen manipulation tasks. High task success does not imply safe execution: high-SR tabletop baselines still leave 13 to 15 percent unsafe-episode rates,and 36 to 56 percent of successful RoboCasa-365 rollouts violate at least one active safety clause. Project page: https://safevla.org.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。