为随机安全强化学习设计可证明安全的算法，提升学习稳定性与可靠性。
Provably Safe Reinforcement Learning for Stochastic Reach-Avoid Problems with Entropy Regularization
- 基于不确定性乐观原则与熵正则化设计安全强化学习算法
- 理论证明算法在有限样本下可实现高概率安全约束，且波动显著降低
- 适合对安全性要求高的在线决策场景，如自动驾驶、机器人控制
我们研究马尔可夫决策过程中的带安全约束最优策略学习问题，采用可达-避让（reach-avoid）设定。目标是设计在线强化学习算法，在学习过程中以任意高概率满足安全约束。首先提出基于不确定性乐观原则（OFU）的算法；在此基础上，提出主算法，引入熵正则化。分析了两个算法的有限样本性能，推导出其后悔边界。结果表明，熵正则化不仅改善了后悔界，还显著控制了传统OFU类安全强化学习算法固有的每轮间波动性。
原文摘要 · Abstract (English)
We consider the problem of learning the optimal policy for Markov decision processes with safety constraints. We formulate the problem in a reach-avoid setup. Our goal is to design online reinforcement learning algorithms that ensure safety constraints with arbitrarily high probability during the learning phase. To this end, we first propose an algorithm based on the optimism in the face of uncertainty (OFU) principle. Based on the first algorithm, we propose our main algorithm, which utilizes entropy regularization. We investigate the finite-sample analysis of both algorithms and derive their regret bounds. We demonstrate that the inclusion of entropy regularization improves the regret and drastically controls the episode-to-episode variability that is inherent in OFU-based safe RL algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。