提出新方法在不确定环境下安全优化策略,迭代次数少于现有方法6倍。
Efficient Policy Optimization in Robust Constrained MDPs with Iteration Complexity Guarantees
- 分阶段优化:先满足约束,再最大化鲁棒奖励
- 仅需 $O(ε^{-2})$ 次迭代即达 $ε$ 精度的可行策略
- 无需二分搜索,计算效率提升4-6倍,适合高风险控制场景
受限决策对设计真实控制系统中的安全策略至关重要,但模拟环境常无法捕捉现实中的不确定性。本文研究在真实模型与可用仿真器/名义模型存在差异时,如何学习最大化累积奖励且满足约束的策略。针对鲁棒约束马尔可夫决策过程(RCMDP),即在围绕未知名义模型的不确定性集中,面对最坏情况随机模型仍需最大化奖励并满足约束的问题,提出一种新方法。由于缺乏强对偶性,传统原对偶方法不适用;同时,因收益与约束函数的最坏模型可能不同,标准鲁棒值迭代法也不适用。本方法能有效最小化约束值函数以满足约束,一旦所有约束满足,则直接最大化鲁棒奖励值函数。理论证明该算法可在 $O(ε^{-2})$ 次迭代内找到 $ε$ 子最优且可行的策略。相比现有最优方法,无需二分搜索,在小折扣因子 $γ$ 下提速至少4倍,在大 $γ$ 时提速至少6倍。
原文摘要 · Abstract (English)
Constrained decision-making is essential for designing safe policies in real-world control systems, yet simulated environments often fail to capture real-world adversities. We consider the problem of learning a policy that will maximize the cumulative reward while satisfying a constraint, even when there is a mismatch between the real model and an accessible simulator/nominal model. In particular, we consider the robust constrained Markov decision problem (RCMDP) where an agent needs to maximize the reward and satisfy the constraint against the worst possible stochastic model under the uncertainty set centered around an unknown nominal model. Primal-dual methods, effective for standard constrained MDP (CMDP), are not applicable here because of the lack of the strong duality property. Further, one cannot apply the standard robust value-iteration based approach on the composite value function either as the worst case models may be different for the reward value function and the constraint value function. We propose a novel technique that effectively minimizes the constraint value function--to satisfy the constraints; on the other hand, when all the constraints are satisfied, it can simply maximize the robust reward value function. We prove that such an algorithm finds a policy with at most $ε$ sub-optimality and feasible policy after $O(ε^{-2})$ iterations. In contrast to the state-of-the-art method, we do not need to employ a binary search, thus, we reduce the computation time by at least 4x for smaller value of discount factor ($γ$) and by at least 6x for larger value of $γ$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。