提出贝尔曼误差中心化方法，提升强化学习算法稳定性
Bellman Error Centering
- 将价值函数更新中的贝尔曼误差进行中心化处理
- 设计了C-TD和CTDC两个算法，证明其收敛性
- 适合需要稳定训练的强化学习场景
本文重新审视了近期提出的奖励中心化算法，指出简单奖励中心化（SRC）确实是奖励中心化，而基于价值的奖励中心化（VRC）本质上是贝尔曼误差中心化（BEC）。基于BEC，我们给出了表格值函数的中心化不动点，以及线性值函数近似下的中心化TD不动点。设计了在策略的CTD算法和离策略的CTDC算法，并证明了两者的收敛性。最后通过实验验证了所提算法的稳定性。贝尔曼误差中心化为多种强化学习算法的扩展提供了支持。
原文摘要 · Abstract (English)
This paper revisits the recently proposed reward centering algorithms including simple reward centering (SRC) and value-based reward centering (VRC), and points out that SRC is indeed the reward centering, while VRC is essentially Bellman error centering (BEC). Based on BEC, we provide the centered fixpoint for tabular value functions, as well as the centered TD fixpoint for linear value function approximation. We design the on-policy CTD algorithm and the off-policy CTDC algorithm, and prove the convergence of both algorithms. Finally, we experimentally validate the stability of our proposed algorithms. Bellman error centering facilitates the extension to various reinforcement learning algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。