arXiv:2606.20698cs.RO2026-06被引 1

用虚拟世界模型让视觉语言动作系统安全地学习,避免真实环境试错。

SafeDojo: Safe Reinforcement Learning for VLA via Interactive World Model

论文配图:SafeDojo: Safe Reinforcement Learning for VLA via Interactive World Model
图 1 · 摘自论文原文
  • 基于交互式视频世界模型进行在线强化学习,通过想象预测未来状态。
  • 在SafeLIBERO上实现最高安全成功率,比最强基线提升8.25个百分点。
  • 适合需要安全部署的机器人任务,尤其适用于开放世界物理环境。

安全控制是实现真实世界具身智能的前提,安全强化学习为此提供了有前景的解决方案。然而,现有方法或需昂贵的真实世界探索,或依赖人工设计的安全函数,均难以扩展至开放世界物理环境中部署的视觉-语言-动作模型。本文提出SafeDojo,首个面向视觉-语言-动作策略的基于模型的安全强化学习框架,通过世界模型生成想象中的未来状态来学习安全行为。SafeDojo在交互式视频世界模型上执行在线强化学习,该模型生成条件于动作的未来预测;通过定制的ResNet成功分类器从想象帧中估计每步任务进展,轻量级安全头则结合潜在上下文与提议的动作块预测每步安全成本,实现任务执行与轨迹安全的同步评估。任务奖励与安全成本信号通过基于拉格朗日约束的GRPO目标平衡,支持在显式约束下协同优化任务成功率与安全性。在SafeLIBERO数据集上,SafeDojo在推理时安全、无模型强化学习与基于模型强化学习基线中表现最佳,综合任务成功率、安全成功率和执行效率均领先,尤其在两级任务中平均安全成功率最优,一级任务上较最强基线提升8.25个百分点。真实世界Franka机械臂部署进一步验证其在五项任务中达到最高平均任务成功率与安全成功率。结果表明,基于世界模型的安全强化学习是一条可扩展且通用的具身智能安全路径。

原文摘要 · Abstract (English)

Safe control is a prerequisite for real-world embodied intelligence, for which safe reinforcement learning has emerged as a promising paradigm. However, existing safe reinforcement learning methods either require costly real-world exploration or depend on hand-crafted safety functions. Neither scales to vision-language-action models deployed in open-world physical environments. We propose SafeDojo, the first model-based safe reinforcement learning framework for vision-language-action policies designed to learn safe actions through world model-based imagination. Specifically, SafeDojo performs online reinforcement learning on top of an interactive video world model. The world model generates action-conditioned future predictions, from which a tailored ResNet success classifier estimates per-step task progress from imagined frames and a lightweight safety head predicts per-step safety costs from latent context together with the proposed action chunk, enabling simultaneous assessment of task execution and trajectory safety. The decoupled task-reward and safety-cost signals are balanced through a Lagrangian-based constrained GRPO objective, enabling coordinated improvement of task success and safety under explicit constraints. On SafeLIBERO, SafeDojo achieves the best aggregate task success, safe success, and execution efficiency among inference-time safety, model-free RL, and model-based RL baselines, with the best average safe-success rate on both levels and an 8.25 percentage-point improvement over the strongest baseline on Level I. Real-world Franka deployment further shows the best average task and safe-success rates across five tasks. Our results position world model-based safe reinforcement learning as a scalable and generalizable path toward safe embodied intelligence.

安全强化学习视觉语言动作世界模型机器人控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。