为连续空间强化学习设计可实现的安全防护机制,防止机器人系统出错。
Realizable Continuous-Space Shields for Safe Reinforcement Learning
- 基于可实现性原理,动态生成安全动作以匹配连续状态与动作空间
- 能处理非马尔可夫安全约束,如避免循环路径,确保始终有解
- 在导航和多智能体环境中验证,既保安全又不降低任务成功率
深度强化学习在多个领域取得显著进展,但缺乏额外保护时仍可能引发灾难性失败。为解决此问题,本文提出首个专为连续状态与动作空间设计的屏蔽方法,确保安全规范的满足,适用于实际机器人应用。该方法基于可实现性——一种保证屏蔽器在任意状态下总能生成安全动作的关键性质,并首次形式化证明了状态相关屏蔽器的可实现性验证能力,从而支持非马尔可夫安全要求(如循环避免)。实验在导航任务和多智能体粒子环境上验证了该方法的有效性,结果显示其在不降低策略成功率的前提下有效保障了安全性。
原文摘要 · Abstract (English)
While Deep Reinforcement Learning (DRL) has achieved remarkable success across various domains, it remains vulnerable to occasional catastrophic failures without additional safeguards. An effective solution to prevent these failures is to use a shield that validates and adjusts the agent's actions to ensure compliance with a provided set of safety specifications. For real-world robotic domains, it is essential to define safety specifications over continuous state and action spaces to accurately account for system dynamics and compute new actions that minimally deviate from the agent's original decision. In this paper, we present the first shielding approach specifically designed to ensure the satisfaction of safety requirements in continuous state and action spaces, making it suitable for practical robotic applications. Our method builds upon realizability, an essential property that confirms the shield will always be able to generate a safe action for any state in the environment. We formally prove that realizability can be verified for stateful shields, enabling the incorporation of non-Markovian safety requirements, such as loop avoidance. Finally, we demonstrate the effectiveness of our approach in ensuring safety without compromising the policy's success rate by applying it to a navigation problem and a multi-agent particle environment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。