提出新算法，实现约束型强化学习的全局收敛与高效解。
Global Convergence for Average Reward Constrained MDPs with Primal-Dual Actor Critic Algorithm
- 采用原-对偶自然演员-评论家框架，兼顾策略优化与约束满足。
- 在已知混合时间时，收敛与约束违反率均达$ ilde{/mathcal{O}}(1/ ext{sqrt}{T})$。
- 理论性能逼近下界，适用于需严格约束的长期决策场景。
本文研究具有通用参数化的无限时域平均奖励约束马尔可夫决策过程（CMDPs）。提出一种原-对偶自然演员-评论家算法，有效处理约束并保证高收敛速度。当学习者已知混合时间 $τ_{\mathrm{mix}}$ 时，算法在长度为 $T$ 的时域内实现全局收敛与约束违反率 $ ilde{\mathcal{O}}(1/\sqrt{T})$。若未知 $τ_{\mathrm{mix}}$，在 $T \geq \tilde{\mathcal{O}}(τ_{\mathrm{mix}}^{2/ε})$ 条件下，可达率 $ ilde{\mathcal{O}}(1/T^{0.5-ε})$。结果匹配马尔可夫决策过程的理论下界，为平均奖励约束MDP的理论研究树立新基准。
原文摘要 · Abstract (English)
This paper investigates infinite-horizon average reward Constrained Markov Decision Processes (CMDPs) with general parametrization. We propose a Primal-Dual Natural Actor-Critic algorithm that adeptly manages constraints while ensuring a high convergence rate. In particular, our algorithm achieves global convergence and constraint violation rates of $\tilde{\mathcal{O}}(1/\sqrt{T})$ over a horizon of length $T$ when the mixing time, $τ_{\mathrm{mix}}$, is known to the learner. In absence of knowledge of $τ_{\mathrm{mix}}$, the achievable rates change to $\tilde{\mathcal{O}}(1/T^{0.5-ε})$ provided that $T \geq \tilde{\mathcal{O}}\left(τ_{\mathrm{mix}}^{2/ε}\right)$. Our results match the theoretical lower bound for Markov Decision Processes and establish a new benchmark in the theoretical exploration of average reward CMDPs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。