改进Q-learning收敛速度，解决过估计问题。
Double Successive Over-Relaxation Q-Learning with an Extension to Deep Reinforcement Learning
- 提出双次超松弛Q-learning，基于样本且无需模型。
- 理论与实验均显示其偏差低于传统方法。
- 适用于小规模表格问题和大规模深度强化学习。
Q-learning是强化学习中广泛使用的算法，但当折扣因子接近1时收敛速度较慢。已有方法如超松弛（SOR）Q-learning通过引入松弛因子加速收敛，但存在两个主要局限：在表格设置下，松弛参数依赖转移概率，导致非完全无模型；且存在过估计偏差。为此，本文提出一种基于样本、无模型的双次超松弛Q-learning算法。理论上和实证上均表明该算法偏差低于原始的SOR Q-learning。此外，在表格设置下，分析了在迭代值有界假设下的收敛性。该算法进一步扩展至大规模问题，结合深度强化学习。表格版本在轮盘赌和网格世界环境中进行了对比测试，深度强化学习版本则在过估计偏差示例及OpenAI Gym环境上进行了验证。
原文摘要 · Abstract (English)
Q-learning is a widely used algorithm in reinforcement learning (RL), but its convergence can be slow, especially when the discount factor is close to one. Successive Over-Relaxation (SOR) Q-learning, which introduces a relaxation factor to speed up convergence, addresses this issue but has two major limitations: In the tabular setting, the relaxation parameter depends on transition probability, making it not entirely model-free, and it suffers from overestimation bias. To overcome these limitations, we propose a sample-based, model-free double SOR Q-learning algorithm. Theoretically and empirically, this algorithm is shown to be less biased than SOR Q-learning. Further, in the tabular setting, the convergence analysis under boundedness assumptions on iterates is discussed. The proposed algorithm is extended to large-scale problems using deep RL. Finally, the tabular version of the proposed algorithm is compared using roulette and grid world environments, while the deep RL version is tested on a maximization bias example and OpenAI Gym environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。