提出新算法解决约束强化学习中的安全策略问题，效果媲美折扣设置。
Primal-Only Actor Critic Algorithm for Robust Constrained Average Cost MDPs
- 仅用原始变量的演员-评论家框架，避开对偶难题
- 在平均成本下实现ε-可行性与ε-最优性，样本复杂度分别约ε⁻⁴和ε⁻⁶
- 适合需要高安全性的工业强化学习场景
本文研究鲁棒约束平均成本马尔可夫决策过程（RCMDPs）中寻找鲁棒且安全策略的问题。该场景面临强对偶性缺失，无法直接使用标准的原始-对偶方法；同时，在平均成本设定下，鲁棒贝尔曼算子在任何范数下均非压缩映射。为此，我们提出一种适用于平均成本RCMDPs的演员-评论家算法。理论证明该方法能同时达到ε-可行性与ε-最优性，并在有无松弛假设下分别具有 ilde{O}(ε^{-4})和 ilde{O}(ε^{-6})的样本复杂度，与折扣情形相当。
原文摘要 · Abstract (English)
In this work, we study the problem of finding robust and safe policies in Robust Constrained Average-Cost Markov Decision Processes (RCMDPs). A key challenge in this setting is the lack of strong duality, which prevents the direct use of standard primal-dual methods for constrained RL. Additional difficulties arise from the average-cost setting, where the Robust Bellman operator is not a contraction under any norm. To address these challenges, we propose an actor-critic algorithm for Average-Cost RCMDPs. We show that our method achieves both \(ε\)-feasibility and \(ε\)-optimality, and we establish a sample complexities of \(\tilde{O}\left(ε^{-4}\right)\) and \(\tilde{O}\left(ε^{-6}\right)\) with and without slackness assumption, which is comparable to the discounted setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。