让强化学习的安全部署决策变得可解释,提升人类信任度。
Explainably Safe Reinforcement Learning

- 用分层决策树表示安全防护机制,实现逐层解释
- 生成的解释树比原防护系统小几个数量级
- 无需额外数据,可无缝集成到现有安全强化学习流程
可信的决策系统需要兼具安全保证和行为可解释性,这对学习型系统尤为重要,因其决策过程常高度不透明。屏蔽法是一种主流的基于模型的安全强化学习技术,但其通过严格形式化方法自动生成,人类难以理解。尽管决策树已成为控制器与策略的常用表达方式,但由于屏蔽机制本质上是非确定性的,其决策树表示在实践中会过大而无法解释。为此,我们提出一种新的可解释安全强化学习方法,通过构建分层决策树来呈现屏蔽策略的决策逻辑,提供自顶向下的案例式解释。设计阶段利用世界模型分析特定状态下执行动作的安全风险,据此构建屏蔽机制及高层决策树,将状态划分为安全、临界、危险、不安全四类,解释为何某情境属于安全临界。运行时生成局部决策树,说明哪些动作被允许,以及为何其他动作被视为不安全。该方法提升了安全强化学习中安全性的可解释性,仅需已有屏蔽信息,开销极低,且易于融入现有屏蔽强化学习流程。实验表明,所生成的解释树比原始屏蔽系统小数个数量级。
原文摘要 · Abstract (English)
Trust in a decision-making system requires both safety guarantees and the ability to interpret and understand its behavior. This is particularly important for learned systems, whose decision-making processes are often highly opaque. Shielding is a prominent model-based technique for enforcing safety in reinforcement learning. However, because shields are automatically synthesized using rigorous formal methods, their decisions are often similarly difficult for humans to interpret. Recently, decision trees became customary to represent controllers and policies. However, since shields are inherently non-deterministic, their decision tree representations become too large to be explainable in practice. To address this challenge, we propose a novel approach for explainable safe RL that enhances trust by providing human-interpretable explanations of the shield's decisions. Our method represents the shielding policy as a hierarchy of decision trees, offering top-down, case-based explanations. At design time, we use a world model to analyze the safety risks of executing actions in given states. Based on this analysis, we construct both the shield and a high-level decision tree that classifies states into risk categories (safe, critical, dangerous, unsafe), explaining why a situation may be safety-critical. At runtime, we generate localized decision trees that explain which actions are allowed and why others are deemed unsafe. Our method facilitates explainability of the safety aspect in safe-by-shielding reinforcement learning, requires no additional information beyond what is already used for shielding, incurs minimal overhead, and integrates readily into existing shielded RL pipelines. In our experiments, we compute explanations using decision trees that are several orders of magnitude smaller than the original shield.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。