提出新算法提升强化学习在不确定环境下的安全约束鲁棒性
Mirror Descent Policy Optimisation for Robust Constrained Markov Decision Processes
- 用镜面下降法同时优化策略与对抗性转移核
- 样本环境下收敛速度达 $\tilde{\mathcal{O}}(1/T^{1/3})$
- 适合需安全保障的机器人、自动驾驶等应用
安全性是强化学习系统的核心要求。新兴的鲁棒约束马尔可夫决策过程框架能学习满足长期约束且在认知不确定性下仍具保障的策略。本文提出针对该框架的镜面下降策略优化方法，利用策略梯度技术同时优化策略（作为最大化者）和转移核（作为对抗最小化者），在表示约束马尔可夫决策过程的拉格朗日函数上进行优化。所提算法在基于样本的鲁棒约束马尔可夫决策过程设置中达到 $\tilde{\mathcal{O}}(1/T^{1/3})$ 的收敛速率。论文还提出一种在转移核空间中的近似梯度下降算法，对一般马尔可夫决策过程设计对抗环境具有独立价值。实验验证了镜面下降策略优化在约束与无约束优化中的优势，相比基线算法在鲁棒性测试中显著提升。
原文摘要 · Abstract (English)
Safety is an essential requirement for reinforcement learning systems. The newly emerging framework of robust constrained Markov decision processes allows learning policies that satisfy long-term constraints while providing guarantees under epistemic uncertainty. This paper presents mirror descent policy optimisation for robust constrained Markov decision processes, making use of policy gradient techniques to optimise both the policy (as a maximiser) and the transition kernel (as an adversarial minimiser) on the Lagrangian representing a constrained Markov decision process. Our proposed algorithm obtains an $\tilde{\mathcal{O}}\left(1/T^{1/3}\right)$ convergence rate in the sample-based robust constrained Markov decision process setting. The paper also contributes an algorithm for approximate gradient descent in the space of transition kernels, which is of independent interest for designing adversarial environments in general Markov decision processes. Experiments confirm the benefits of mirror descent policy optimisation in constrained and unconstrained optimisation, and significant improvements are observed in robustness tests when compared to baseline policy optimisation algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。