arXiv:2606.03804cs.LG2026-06

让强化学习安全探索像调用接口一样简单,无需懂形式化方法。

Easy-to-Use Shielding for Reinforcement Learning

论文配图:Easy-to-Use Shielding for Reinforcement Learning
图 1 · 摘自论文原文
  • 通过 Python 库 tempestpy 将形式化安全防护集成到 Gymnasium 环境中
  • 在多个环境中验证,使用防护的智能体成功率提升 20%~40% 且零危险行为
  • 专为初学者设计,适合想快速应用安全探索的 RL 实践者

安全探索是强化学习中的关键挑战,旨在防止智能体在探索过程中做出有害决策。屏蔽(Shielding)是一种利用环境模型作为领域知识来判断动作安全性的技术。尽管已有成熟理论,但因缺乏与标准强化学习框架无缝衔接的端到端基础设施,其应用受限。通常需具备形式化方法专业知识和大量工程投入,难以融入常规强化学习流程。为此,我们扩展了 shield 合成工具 Tempest,开发出 tempestpy:一个直接集成于 Gymnasium API 的 Python 库,支持在现有强化学习流水线中合成并部署屏蔽。该工作大幅降低使用门槛,使形式化安全探索方法变为可实际使用的组件。我们还拓展 Tempest 算法以支持随机多人博弈场景,保证形式化安全承诺。实验展示了完整工作流,并在多个环境评估了有无屏蔽的强化学习表现。为便于建模,我们提供了 MiniGrid 的符号化模型,并引入 MiniGridSafe——一组面向安全设计的测试环境,包含概率转移和多智能体,可在简洁直观的设置中研究复杂安全问题。

原文摘要 · Abstract (English)

Safe exploration is a key challenge in Reinforcement Learning (RL) that aims to prevent agents from making harmful decisions while exploring their environment. Safe exploration is a key challenge in Reinforcement Learning (RL) that aims to prevent agents from making harmful decisions while exploring their environment. Shielding is one such technique that assumes domain knowledge in the form of an environment model to decide upon action safety. Although well-established, shielding has seen limited adoption in RL due to the lack of accessible end-to-end infrastructure connecting formal shield synthesis with standard RL frameworks. Applying shielding typically requires expertise in formal methods and substantial engineering effort, keeping it outside the typical RL workflow. We address this by extending our shield synthesis tool Tempest into a practical backend for safe RL. Our core contribution is tempestpy, a Python library that integrates Tempest-based shield synthesis directly into the Gymnasium API, allowing shields to be synthesized and deployed within existing RL pipelines. This lowers the barrier to entry for shielding and turns formal safe-exploration methods into a usable component for RL practitioners. We also extend Tempest's algorithmic support to compute sound shields for stochastic multiplayer games, preserving formal safety guarantees. We demonstrate the resulting workflow end to end and evaluate shielded and unshielded RL across multiple environments. To facilitate modeling, we provide symbolic models for MiniGrid and introduce MiniGridSafe, a collection of playground environments designed to make shielding easily accessible and experimentally transparent. MiniGridSafe extends MiniGrid with safety-oriented scenarios featuring probabilistic transitions and additional agents, enabling the study of challenging safety aspects in a simple and intuitive setting.

强化学习安全探索形式化方法工具库

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。