提出新算法让强化学习在函数逼近下仍能稳定收敛。
Monotone and Conservative Policy Iteration Beyond the Tabular Case
- 用约束优化保证价值估计单调且低于真实回报
- 新算法在模拟中优于传统方法，每步都有性能提升保障
- 适合需要可靠收敛的复杂强化学习场景
我们提出可靠策略迭代（RPI）和保守RPI（CRPI），在函数逼近下保留表格情形下的理论保证。RPI采用新的贝尔曼约束优化进行策略评估，恢复了价值估计的单调性，并证明其值始终下界于真实回报；极限情况下部分满足未投影的贝尔曼方程。CRPI共享相同的评估机制，但通过最大化显式考虑函数逼近误差的性能差下界来保守更新策略，继承了RPI的所有保证，并具备每步改进的理论边界。初步仿真显示，RPI与CRPI优于传统策略迭代及其变体。本工作填补了强化学习中的基础空白：如TRPO和PPO等流行算法源自表格型保守策略迭代（CPI），但在函数逼近下使用时，其理论保证常失效，导致发散、振荡或收敛至次优策略。RPI与CRPI为任意函数类恢复了策略迭代/保守策略迭代风格的保证，为下一代强化学习提供了严谨基础。
原文摘要 · Abstract (English)
We introduce Reliable Policy Iteration (RPI) and Conservative RPI (CRPI), variants of Policy Iteration (PI) and Conservative PI (CPI), that retain tabular guarantees under function approximation. RPI uses a novel Bellman-constrained optimization for policy evaluation. We show that RPI restores the textbook \textit{monotonicity} of value estimates and that these estimates provably \textit{lower-bound} the true return; moreover, their limit partially satisfies the \textit{unprojected} Bellman equation. CRPI shares RPI's evaluation, but updates policies conservatively by maximizing a new performance-difference \textit{lower bound} that explicitly accounts for function-approximation-induced errors. CRPI inherits RPI's guarantees and, crucially, admits per-step improvement bounds. In initial simulations, RPI and CRPI outperform PI and its variants. Our work addresses a foundational gap in RL: popular algorithms such as TRPO and PPO derive from tabular CPI yet are deployed with function approximation, where CPI's guarantees often fail-leading to divergence, oscillations, or convergence to suboptimal policies. By restoring PI/CPI-style guarantees for \textit{arbitrary} function classes, RPI and CRPI provide a principled basis for next-generation RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。