提出可采样的参数化策略，实现线性马尔可夫决策过程的高效探索与最优采样复杂度。
Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 使用对数线性参数化策略和可计算的logit匹配目标
- 在策略内和策略外设置下分别达到ε⁻⁴和ε⁻²的样本复杂度
- 适合关注理论与实践平衡的强化学习研究者
尽管演员-评论家方法在实践中表现良好，但其理论分析存在局限：现有工作或通过强假设回避探索问题，或分析算法复杂修改后的不切实际方法。此外，针对线性马尔可夫决策过程（MDP）的演员-评论家方法常采用自然策略梯度并构造无显式参数化的“隐式”策略，导致采样计算开销大，环境交互效率低。为此，本文聚焦有限时域线性MDP，提出一种使用参数化对数线性策略的乐观演员-评论家框架。演员部分引入可计算的“logit匹配”回归目标；评论家部分通过朗之万蒙特卡洛近似汤普森采样以获得乐观价值估计。证明该算法在策略内和策略外设置下分别达到˜O(ε⁻⁴)和˜O(ε⁻²)的样本复杂度。结果与已有理论工作相比达到了最优样本复杂度，且算法更贴近实际应用。
原文摘要 · Abstract (English)
Although actor-critic methods have been successful in practice, their theoretical analyses have several limitations. Specifically, existing theoretical work either sidesteps the exploration problem by making strong assumptions or analyzes impractical methods with complicated algorithmic modifications. Moreover, the actor-critic methods analyzed for linear MDPs often employ natural policy gradient and construct "implicit" policies without explicit parameterization. Such policies are computationally expensive to sample from, making the environment interactions inefficient. To that end, we focus on the finite-horizon linear MDPs and propose an optimistic actor-critic framework that uses parametric log-linear policies. In particular, we introduce a tractable $\textit{logit-matching}$ regression objective for the actor. For the critic, we use approximate Thompson sampling via Langevin Monte Carlo to obtain optimistic value estimates. We prove that the resulting algorithm achieves $\widetilde{\mathcal{O}}(ε^{-4})$ and $\widetilde{\mathcal{O}}(ε^{-2})$ sample complexity in the on-policy and off-policy setting, respectively. Our results match prior theoretical work in achieving the state-of-the-art sample complexity, while our algorithm is more aligned with practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。