发现线性近似下软最大策略梯度可全局收敛，无需依赖近似误差。
Rethinking the Global Convergence of Softmax Policy Gradient with Linear Function Approximation
- 基于线性特征表示，证明软最大策略梯度可全局收敛。
- 使用特定学习率时，收敛速度达 O(1/T)。
- 任意固定学习率也能保证收敛，适合理论研究者。
策略梯度（PG）方法在强化学习的实践中取得了显著成功。为处理大规模状态-动作空间，通常需结合函数逼近。在此设定下，建模问题相关量的近似误差是刻画PG方法全局收敛性的关键因素。本文聚焦于带线性函数逼近的软最大策略梯度（记为\texttt{Lin-SPG}），证明即使在随机老虎机场景中，该算法的全局收敛性也与近似误差无关。我们首次给出了\texttt{Lin-SPG}实现渐近全局收敛的充要特征表示条件。在满足这些条件的前提下，我们证明：使用问题相关学习率时，\texttt{Lin-SPG}经过T次迭代可达到O(1/T)的收敛速率；此外，任意固定学习率同样能确保渐近全局收敛至最优策略。
原文摘要 · Abstract (English)
Policy gradient (PG) methods have played an essential role in the empirical successes of reinforcement learning. In order to handle large state-action spaces, PG methods are typically used with function approximation. In this setting, the approximation error in modeling problem-dependent quantities is a key notion for characterizing the global convergence of PG methods. We focus on Softmax PG with linear function approximation (referred to as $\texttt{Lin-SPG}$) and demonstrate that the approximation error is irrelevant to the algorithm's global convergence even for the stochastic bandit setting. Consequently, we first identify the necessary and sufficient conditions on the feature representation that can guarantee the asymptotic global convergence of $\texttt{Lin-SPG}$. Under these feature conditions, we prove that $T$ iterations of $\texttt{Lin-SPG}$ with a problem-specific learning rate result in an $O(1/T)$ convergence to the optimal policy. Furthermore, we prove that $\texttt{Lin-SPG}$ with any arbitrary constant learning rate can ensure asymptotic global convergence to the optimal policy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。