为强化学习设计可预测的安全防护罩，兼顾安全与性能。
Predictive Safety Shield for Dyna-Q Reinforcement Learning
- 基于环境模型进行安全仿真，动态更新局部Q值
- 短预测时域即可找到最优路径，提升决策效率
- 对仿真与现实差异鲁棒，无需额外训练
强化学习的安全保障是其实现真实应用的主要挑战。安全防护罩扩展了标准强化学习，实现硬性安全保证。然而，现有防护罩通常采用随机采样安全动作或固定备用控制器，忽略不同安全动作对未来性能的影响。本文提出一种面向离散空间模型化强化学习代理的预测性安全防护罩。该防护罩基于安全仿真产生的预测，局部更新Q函数。此方法在保持硬性安全保证的同时提升了性能。网格世界实验表明，即使预测时域较短，也能识别出最优路径。且该方法对分布偏移（如仿真与现实之间）具有鲁棒性，无需额外训练。
原文摘要 · Abstract (English)
Obtaining safety guarantees for reinforcement learning is a major challenge to achieve applicability for real-world tasks. Safety shields extend standard reinforcement learning and achieve hard safety guarantees. However, existing safety shields commonly use random sampling of safe actions or a fixed fallback controller, therefore disregarding future performance implications of different safe actions. In this work, we propose a predictive safety shield for model-based reinforcement learning agents in discrete space. Our safety shield updates the Q-function locally based on safe predictions, which originate from a safe simulation of the environment model. This shielding approach improves performance while maintaining hard safety guarantees. Our experiments on gridworld environments demonstrate that even short prediction horizons can be sufficient to identify the optimal path. We observe that our approach is robust to distribution shifts, e.g., between simulation and reality, without requiring additional training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。