动态调整训练周期，让策略梯度更快收敛到最优解。
Structure Matters: Dynamic Policy Gradient
- 将无限时域问题拆成一系列上下文老虎机问题迭代求解。
- 在表格型软最大参数化下，收敛速度与有效时长远程相关，呈多项式增长。
- 适合研究强化学习收敛性或改进策略梯度方法的研究者。
本文研究γ-折扣的无限时域表格型马尔可夫决策过程（MDPs），提出动态策略梯度（DynPG）框架。该框架将动态规划与任意策略梯度方法结合，显式利用环境的马尔可夫特性。DynPG在训练中动态调整问题时域，将原无限时域MDP分解为一系列上下文老虎机问题。通过迭代求解这些上下文老虎机问题，DynPG收敛至无限时域MDP的稳定最优策略。为验证其有效性，我们在表格型软最大参数化下建立了非渐近全局收敛速率，关注关键参数对结果的影响。结合经典动态规划分析与近期策略梯度收敛论证，证明了软最大版DynPG在有效时远（1−γ）⁻¹上呈多项式尺度。这一发现与最近对原始策略梯度的指数下界形成对比。
原文摘要 · Abstract (English)
In this work, we study $γ$-discounted infinite-horizon tabular Markov decision processes (MDPs) and introduce a framework called dynamic policy gradient (DynPG). The framework directly integrates dynamic programming with (any) policy gradient method, explicitly leveraging the Markovian property of the environment. DynPG dynamically adjusts the problem horizon during training, decomposing the original infinite-horizon MDP into a sequence of contextual bandit problems. By iteratively solving these contextual bandits, DynPG converges to the stationary optimal policy of the infinite-horizon MDP. To demonstrate the power of DynPG, we establish its non-asymptotic global convergence rate under the tabular softmax parametrization, focusing on the dependencies on salient but essential parameters of the MDP. By combining classical arguments from dynamic programming with more recent convergence arguments of policy gradient schemes, we prove that softmax DynPG scales polynomially in the effective horizon $(1-γ)^{-1}$. Our findings contrast recent exponential lower bound examples for vanilla policy gradient.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。