让AI看视频实时预警危险动作，还能理解上下文判断是否安全。
VLESA: Vision-Language Embodied Safety Agent for Human Activity Monitoring
- 用视觉语言模型分析第一视角视频，结合任务目标判断动作安全性。
- 在ASIMOV-2.0上干预准确率达93.6%，安全提升超41个百分点。
- 适合人机协作、工业安全监控等需要实时风险预警的场景。
随着AI系统越来越多地参与人类物理任务，安全性至关重要——物理动作具有即时且不可逆的后果，不同于数字错误。我们提出视觉语言具身安全代理（VLESA），通过第一视角视频监测人类活动，并在预测到危险行为时触发实时干预。VLESA解决了依赖意图的安全性问题：相同动作在不同上下文中可能安全或危险。为此，我们构建了一个数据集，将第一视角图像帧与目标条件化的安全标注配对，从而训练一个基于GRPO的目标条件化安全Q滤波器，可评估动作相对于推断意图的安全性，无需重新训练。此外，还提出一个意图-动作预测代理，能从视频中联合推断目标并预测未来动作。在ASIMOV-2.0基准测试中，VLESA在真实标签帧上的干预准确率达到93.6%，优于基线；而经过GRPO训练的Q滤波器通过目标条件化约束解码，使动作安全性提升超过41个百分点。代码已开源：https://github.com/HanjiangHu/VLESA。
原文摘要 · Abstract (English)
As AI systems increasingly assist humans in physical tasks, ensuring safety becomes paramount -- physical actions carry immediate and irreversible consequences that digital errors do not. We introduce the Vision-Language Embodied Safety Agent (VLESA), a framework that monitors human activities from egocentric video and triggers real-time safety interventions when dangerous actions are predicted. VLESA addresses intent-dependent safety where identical actions can be safe or dangerous depending on context. A dataset pairing egocentric frames with goal-conditioned safety annotations is introduced, enabling a goal-conditioned safety Q-filter trained via GRPO that evaluates actions with respect to inferred intent without retraining. On top of that, an intent-action prediction agent is proposed to jointly infer goals and predict future actions from video. On the ASIMOV-2.0 benchmark, VLESA achieves higher intervention accuracy at the exact ground-truth frame compared to baselines, while the GRPO-trained Q-filter improves action safety by over 41 percentage points through goal-conditioned constrained decoding. Code is available at https://github.com/HanjiangHu/VLESA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。