首个适用于上下文MDP的近最优策略优化算法，理论表现更优。
Near-Optimal Regret for Policy Optimization in Contextual MDPs with General Offline Function Approximation
- 基于乐观策略优化框架，在离线函数逼近下求解上下文MDP。
- 首次实现状态、动作空间规模的最优依赖关系，理论性能超越现有方法。
- 适合研究强化学习理论与离线策略优化的学者参考。
我们提出 exttt{OPO-CMDP}，首个在一般离线函数逼近下针对随机上下文马尔可夫决策过程（CMDPs）的策略优化算法。该方法在高概率下达到 $ ilde{O}(H^4 oot{4}{T|S||A|"log(| F|| P|)}$ 的后悔界，其中 $S$、$A$ 分别为状态与动作空间，$H$ 为时域长度，$T$ 为回合数，$ F$、$ P$ 为用于逼近损失和动态的有限函数类。这是首个在 $|S|$ 与 $|A|$ 上具有最优依赖性的后悔界，直接改进了当前最优结果（Qian, Hu, and Simchi-Levi, 2024）。结果表明，乐观策略优化为求解 CMDPs 提供了一条自然、计算高效且理论近最优的路径。
原文摘要 · Abstract (English)
We introduce \texttt{OPO-CMDP}, the first policy optimization algorithm for stochastic Contextual Markov Decision Process (CMDPs) under general offline function approximation. Our approach achieves a high probability regret bound of $\widetilde{O}(H^4\sqrt{T|S||A|\log(|\mathcal{F}||\mathcal{P}|)}),$ where $S$ and $A$ denote the state and action spaces, $H$ the horizon length, $T$ the number of episodes, and $\mathcal{F}, \mathcal{P}$ the finite function classes used to approximate the losses and dynamics, respectively. This is the first regret bound with optimal dependence on $|S|$ and $|A|$, directly improving the current state-of-the-art (Qian, Hu, and Simchi-Levi, 2024). These results demonstrate that optimistic policy optimization provides a natural, computationally superior and theoretically near-optimal path for solving CMDPs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。