首次给出约束平均奖励MDP的最优样本复杂度,解决理论空白。
Near-Optimal Sample Complexity Bounds for Constrained Average-Reward MDPs
- 设计基于模型的算法,分宽松与严格可行性两种场景处理约束。
- 严格可行下样本复杂度达 $ ilde{O} ig(rac{S A (B+H)}{ε^2 ζ^2}ig)$,与下界匹配。
- 适用于需长期满足约束的强化学习任务,如能源调度、安全控制。
近期研究显著推进了对生成模型下平均奖励马尔可夫决策过程(AMDP)学习样本复杂度的理解。然而,对需满足长期平均约束的约束平均奖励MDP(CAMDP)了解仍有限。本文研究在生成模型下学习ε-最优策略的样本复杂度。提出一种基于模型的算法,适用于两种设定:(i) 容许小量约束违反的宽松可行性;(ii) 输出策略严格满足约束的严格可行性。证明算法在宽松和严格可行性下分别达到$ ilde{O}ig(rac{S A (B+H)}{ ε^2}ig)$ 和 $ ilde{O} ig(rac{S A (B+H)}{ε^2 ζ^2}ig)$ 的样本复杂度。其中ζ为Slater常数,反映可行域大小;H为偏置函数的跨度界;B为瞬态时间界。进一步建立严格可行性下的匹配下界$ ildeΩig(rac{S A (B+H)}{ ε^2ζ^2}ig)$,首次实现CAMDP的极小极大最优边界。本工作填补了约束平均奖励MDP理论理解的空白。
原文摘要 · Abstract (English)
Recent advances have significantly improved our understanding of the sample complexity of learning in average-reward Markov decision processes (AMDPs) under the generative model. However, much less is known about the constrained average-reward MDP (CAMDP), where policies must satisfy long-run average constraints. In this work, we address this gap by studying the sample complexity of learning an $ε$-optimal policy in CAMDPs under a generative model. We propose a model-based algorithm that operates under two settings: (i) relaxed feasibility, which allows small constraint violations, and (ii) strict feasibility, where the output policy satisfies the constraint. We show that our algorithm achieves sample complexities of $\tilde{O}\left(\frac{S A (B+H)}{ ε^2}\right)$ and $\tilde{O} \left(\frac{S A (B+H)}{ε^2 ζ^2} \right)$ under the relaxed and strict feasibility settings, respectively. Here, $ζ$ is the Slater constant indicating the size of the feasible region, $H$ is the span bound of the bias function, and $B$ is the transient time bound. Moreover, a matching lower bound of $\tildeΩ\left(\frac{S A (B+H)}{ ε^2ζ^2}\right)$ for the strict feasibility case is established, thus providing the first minimax-optimal bounds for CAMDPs. Our results close the theoretical gap in understanding the complexity of constrained average-reward MDPs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。