用不理想轨迹训练模型避开危险行为，提升安全性和性能。
SafeMIL: Learning Offline Safe Imitation Policy from Non-Preferred Trajectories
- 通过多实例学习构建风险预测成本函数，识别危险状态动作对。
- 在不降低奖励表现的前提下满足安全约束，优于多个基线方法。
- 适合需要高安全性的离线强化学习场景，如自动驾驶、医疗决策。
本文研究离线安全模仿学习问题。在许多现实场景中，在线交互存在风险，且难以精确指定每一步的奖励与安全成本。然而，收集反映不良或危险行为的轨迹通常可行，这些轨迹隐含了应避免的行为模式，我们称之为非优选轨迹。不同于标准模仿学习仅模仿示范行为，本工作要求智能体利用非优选轨迹学习规避风险行为。为此，我们提出SafeMIL方法，通过多实例学习（Multiple Instance Learning）构建参数化成本函数，以预测状态-动作对是否具有风险。该成本函数用于引导策略学习，使最终策略优先考虑安全性。实验表明，所提方法可在不牺牲奖励性能的前提下满足成本约束，显著优于多个基线方法。
原文摘要 · Abstract (English)
In this work, we study the problem of offline safe imitation learning (IL). In many real-world settings, online interactions can be risky, and accurately specifying the reward and the safety cost information at each timestep can be difficult. However, it is often feasible to collect trajectories reflecting undesirable or risky behavior, implicitly conveying the behavior the agent should avoid. We refer to these trajectories as non-preferred trajectories. Unlike standard IL, which aims to mimic demonstrations, our agent must also learn to avoid risky behavior using non-preferred trajectories. In this paper, we propose a novel approach, SafeMIL, to learn a parameterized cost that predicts if the state-action pair is risky via Multiple Instance Learning. The learned cost is then used to avoid non-preferred behaviors, resulting in a policy that prioritizes safety. We empirically demonstrate that our approach can learn a safer policy that satisfies cost constraints without degrading the reward performance, thereby outperforming several baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。