自然策略梯度可看作双重平滑的策略迭代，统一了多种强化学习算法。
Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework
- 提出双重平滑策略迭代框架，用加权平均Q函数和正则化贪心更新策略。
- 证明无需额外正则或自适应步长即可实现几何收敛，复杂度为O((1-γ)^{-1} log((1-γ)^{-1}ε^{-1}))。
- 适用于线性函数近似和随机最短路径问题，适合研究理论收敛性的学者。
本文表明，强化学习中的自然策略梯度可精确表示为一种双重平滑且平均化的策略迭代形式。我们提出了双重平滑策略迭代（DSPI）——一种基于贝尔曼算子的框架，其中每个策略通过在历史Q函数加权平均上执行正则化贪心步骤得到。DSPI包含策略迭代、双平均策略迭代、自然策略梯度及更一般的策略双平均方法等作为特例。仅依赖于平滑贝尔曼算子的单调性和压缩性，我们证明了DSPI在分布无关条件下具有全局几何收敛性。因此，标准自然策略梯度与策略双平均方法在不修改MDP、不引入超出更新中固有镜像映射的正则化、也不使用自适应轨迹相关步长的情况下，达到迭代复杂度O((1−γ)^{−1}log((1−γ)^{−1}ε^{−1}))以求得ε-最优策略。对于无正则化贪心情形（对应双平均策略迭代），我们也证明了有限终止性。该贝尔曼算子框架还可推广至带线性函数近似的折扣MDP和随机最短路径问题。
原文摘要 · Abstract (English)
In this work, we show that natural policy gradient, a core algorithm in reinforcement learning, admits an exact formulation as a smoothed and averaged form of policy iteration. Specifically, we introduce doubly smoothed policy iteration (DSPI), a Bellman-operator framework in which each policy is obtained by applying a regularized greedy step to a weighted average of past $Q$-functions. DSPI includes policy iteration, dual-averaged policy iteration, natural policy gradient, and more general policy dual averaging methods as special cases. Using only monotonicity and contraction of smoothed Bellman operators, we prove distribution-free global geometric convergence of DSPI. Consequently, standard natural policy gradient and policy dual averaging achieve an iteration complexity of $\mathcal{O}((1-γ)^{-1}\log((1-γ)^{-1}ε^{-1}))$ for computing an $ε$-optimal policy, without modifying the MDP, adding regularization beyond the mirror map inherent in the update, or using adaptive, trajectory-dependent stepsizes. For the unregularized greedy case, corresponding to dual-averaged policy iteration, we also prove finite termination. The same Bellman-operator framework further extends to discounted MDPs with linear function approximation and stochastic shortest path problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。