提出安全策略更新方法,确保强化学习在持续适应中不丢失安全性能。
SafeAdapt: Provably Safe Policy Updates in Deep Reinforcement Learning
- 引入Rashomon集,在参数空间中定义安全区域,实现事前安全保证。
- 实测在冻湖和毒苹果任务中,更新后仍100%保持原始安全约束。
- 适合需要高可靠性部署的自动驾驶、医疗等安全敏感场景。
安全是强化学习在安全关键任务中部署的先决条件。实际应用中环境动态变化或目标调整,常需更新已学策略,但如何在更新过程中维持原有任务的安全性成为核心挑战。现有方法多缺乏形式化保障,或仅事后验证安全性。本文提出一种全新的事前安全策略更新方法,通过引入Rashomon集——一个在演示数据分布下满足安全约束的策略参数区域——实现对任意强化学习算法的更新投影,从而提供形式化、可证明的安全保障。我们在网格世界导航任务(冻湖与毒苹果)中验证该方法:在下游适应过程中,始终能保证源任务上的确定性安全。相比之下,基于正则化的基线方法出现安全约束的灾难性遗忘,而本方法实现了强适应能力且安全性质严格保留。
原文摘要 · Abstract (English)
Safety guarantees are a prerequisite to the deployment of reinforcement learning (RL) agents in safety-critical tasks. Often, deployment environments exhibit non-stationary dynamics or are subject to changing performance goals, requiring updates to the learned policy. This leads to a fundamental challenge: how to update an RL policy while preserving its safety properties on previously encountered tasks? The majority of current approaches either do not provide formal guarantees or verify policy safety only a posteriori. We propose a novel a priori approach to safe policy updates in continual RL by introducing the Rashomon set: a region in policy parameter space certified to meet safety constraints within the demonstration data distribution. We then show that one can provide formal, provable guarantees for arbitrary RL algorithms used to update a policy by projecting their updates onto the Rashomon set. Empirically, we validate this approach across grid-world navigation environments (Frozen Lake and Poisoned Apple) where we guarantee an a priori provably deterministic safety on the source task during downstream adaptation. In contrast, we observe that regularisation-based baselines experience catastrophic forgetting of safety constraints while our approach enables strong adaptation with provable guarantees that safety is preserved.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。