解释了为何强化学习策略梯度在无折扣场景下依然有效
Why Policy Gradient Algorithms Work for Undiscounted Total-Reward MDPs
- 通过区分状态的常返与暂态特性，构建新分析框架
- 提出暂态访问度量替代传统定义，解决γ=1时的理论难题
- 为大模型强化学习提供理论支持，适合关注理论严谨性的研究者
经典策略梯度方法是现代基于策略的强化学习算法的理论基础。大多数严格的分析（尤其是收敛性证明）都假设折扣因子 $γ< 1$。然而，近期针对大语言模型的基于策略的强化学习工作采用无折扣总回报设定（$γ=1$），导致现有理论难以适用。本文针对无折扣期望总回报无限时域马尔可夫决策过程（MDP），基于两个关键洞察提供了策略梯度方法的分析：(i) 在所有动作概率严格为正的策略集合中（如深度强化学习中使用softmax输出层的模型），状态被划分为常返与暂态的状态类型保持不变；(ii) 当 $γ=1$ 时，传统状态访问度量可能未定义，可被我们提出的“暂态访问度量”替代。
原文摘要 · Abstract (English)
The classical policy gradient method is the theoretical and conceptual foundation of modern policy-based reinforcement learning (RL) algorithms. Most rigorous analyses of such methods, particularly those establishing convergence guarantees, assume a discount factor $γ< 1$. In contrast, however, a recent line of work on policy-based RL for large language models uses the undiscounted total-reward setting with $γ= 1$, rendering much of the existing theory inapplicable. In this paper, we provide analyses of the policy gradient method for undiscounted expected total-reward infinite-horizon MDPs based on two key insights: (i) the classification of the MDP states into recurrent and transient states is invariant over the set of policies that assign strictly positive probability to every action (as is typical in deep RL models employing a softmax output layer) and (ii) the classical state visitation measure (which may be ill-defined when $γ= 1$) can be replaced with a new object that we call the transient visitation measure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。