解析PPO-Clip算法的收敛性，证明其能快速找到最优策略。
Non-Asymptotic Global Convergence of PPO-Clip
- 基于softmax参数化和f-散度正则，建立非均匀光滑性条件
- 对前向KL正则实现非渐近线性收敛，确保全局最优
- 适用于关注强化学习理论保证的研究者或模型优化工程师
强化学习在通过人类反馈强化学习（RLHF）对齐大语言模型方面受到关注。仅更新策略的近端策略优化（PPO）变体因其高效性被广泛采用，其通过裁剪机制提升稳定性，并引入如反向KL散度或更一般的f-散度正则项以防止策略漂移。尽管这些方法在实践中表现良好，但对其问题本质与算法性质的严格理论理解仍有限。本文在一般强化学习设定下，针对softmax策略参数化的演员仅更新型PPO算法，结合f-散度正则，推导出非均匀Lipschitz光滑性条件与Łojasiewicz不等式。基于此，对前向KL正则项建立了非渐近线性收敛至全局最优策略的结果；对反向KL正则项则推导出驻点收敛与局部线性收敛性。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has gained attention for aligning large language models (LLMs) via reinforcement learning from human feedback (RLHF). The actor-only variants of Proximal Policy Optimization (PPO) are widely applied for their efficiency. These algorithms incorporate a clipping mechanism to improve stability. Besides, a regularization term, such as the reverse KL-divergence or a more general \(f\)-divergence, is introduced to prevent policy drift. Despite their empirical success, a rigorous theoretical understanding of the problem and the algorithm's properties is limited. This paper advances the theoretical foundations of the PPO-Clip algorithm by analyzing a deterministic actor-only PPO algorithm within the general RL setting with \(f\)-divergence regularization under the softmax policy parameterization. We derive a non-uniform Lipschitz smoothness condition and a Łojasiewicz inequality for the considered problem. Based on these, a non-asymptotic linear convergence rate to the globally optimal policy is established for the forward KL-regularizer. Furthermore, stationary convergence and local linear convergence are derived for the reverse KL-regularizer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。