通过自动生成安全分支对，让智能体学会在不牺牲任务成功率的前提下规避危险。
SafeBranch: Branch-Pair Safety Alignment for Embodied Agents
- 从不安全轨迹中回滚到违规点，生成仅在该点不同的安全与不安全动作对。
- 在未见物体场景下，安全成功次数比基线高约10倍，且不影响任务完成率。
- 无需部署评判器，适合需要高可靠性的现实交互场景应用。
基于视觉语言模型的具身智能体虽能完成指令任务，但常在过程中违反安全约束，这一问题被定义为交互安全。训练智能体实现安全行为困难，因安全与任务成功是不同目标，且安全问题仅出现在轨迹中的少数关键步骤。标准监督不足：模仿安全轨迹仅教行为而无原因解释，对比任意安全与不安全轨迹会混入无关差异。我们提出SafeBranch框架，通过环境回滚从智能体自身不安全采样轨迹构建分支对。将每条不安全轨迹回滚至引发违规的关键步骤，查询智能体生成安全替代动作，使原始动作与替代动作仅在该步不同。训练后智能体在部署时无需评判器即可安全执行。在IS-Bench、SafetyALFRED及未见任务与物体的分布外变体上，均能可靠处理安全问题，且不降低任务成功率，在未见物体变体上安全成功次数约为未训练基线的十倍。
原文摘要 · Abstract (English)
Vision-language-model-based embodied agents can complete instructed tasks but often violate safety constraints in the process, a problem recently framed as interactive safety. Training such agents to act safely is difficult, since safety and task success are distinct objectives, and safety arises only at a small number of safety-critical steps within a trajectory. Standard supervision is insufficient: imitating safe trajectories teaches behavior without explaining why it is safe, and contrasting arbitrary safe and unsafe trajectories mixes the safety signal with unrelated differences. We propose SafeBranch, a framework that aligns an embodied actor on safety through branch pairs constructed from the actor's own unsafe rollouts via environment rollback. SafeBranch rolls each unsafe rollout back to the safety-critical step that caused the violation, queries the actor for a safe alternative, and pairs the original action with the alternative so that the two branches differ only at that step. The trained actor acts safely at deployment with no critic in the loop. On IS-Bench, SafetyALFRED, and out-of-distribution variants with unseen tasks and objects, it handles safety reliably without sacrificing task success, achieving roughly ten times more safe successes than the untrained baseline on the unseen-object variant.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。