提出一种自适应分布双Q学习方法，有效缓解强化学习中的过估计问题。
ADDQ: Adaptive Distributional Double Q-Learning
- 基于分布强化学习，引入局部自适应过估计控制机制。
- 在表格、Atari和MuJoCo环境中均显著降低过估计并加速收敛。
- 可无缝集成到现有分布算法中，仅需少量代码修改。
Q值估计中的偏差问题是阻碍Q-learning与演员-评论家方法收敛的常见障碍。现代强化学习算法成功的一部分原因在于其直接或间接的过估计抑制机制。本文提出一种基于分布强化学习（DRL）的简单易实现方法，以局部自适应方式缓解过估计问题。该框架易于部署，现有分布算法仅需少量代码即可改进。通过双Q学习提供理论支持，说明如何将局部自适应过估计控制嵌入现有算法。实验覆盖表格型、Atari和MuJoCo环境，验证了方法的有效性。
原文摘要 · Abstract (English)
Bias problems in the estimation of $Q$-values are a well-known obstacle that slows down convergence of $Q$-learning and actor-critic methods. One of the reasons of the success of modern RL algorithms is partially a direct or indirect overestimation reduction mechanism. We propose an easy to implement method built on top of distributional reinforcement learning (DRL) algorithms to deal with the overestimation in a locally adaptive way. Our framework is simple to implement, existing distributional algorithms can be improved with a few lines of code. We provide theoretical evidence and use double $Q$-learning to show how to include locally adaptive overestimation control in existing algorithms. Experiments are provided for tabular, Atari, and MuJoCo environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。