提出双网络Q学习算法，减少深度强化学习中的过估计问题。
Deep Double Q-learning
- 显式训练两个独立的Q函数，解耦动作选择与评估
- 在57个Atari游戏中平均表现优于Double DQN，47个游戏胜出
- 通过共享层、低重放缓冲区比例等设计提升训练稳定性
Double Q-learning 是一种经典控制算法，可缓解Q-learning中的最大化偏差。它通过显式训练两个独立的动作价值函数，在计算自举目标时解耦动作选择与评估。Double DQN 将目标自举解耦引入深度强化学习，但仅训练单个动作价值函数，未能完全解耦估计器，导致两个估计器仍存在相关性，过估计问题依然存在。本文提出深度双Q学习（DDQL），一种显式训练两个Q函数的深度强化学习算法。通过降低重放缓冲区比例、延长目标网络更新周期以及共享网络层等技术组合，DDQL 稳定了训练过程。在57个Atari 2600游戏上，DDQL 的综合表现优于Double DQN，有47个游戏实现超越，同时进一步降低了过估计程度。此外，我们研究了将Double Q-learning适配到深度强化学习的关键设计选择，包括网络结构、重放缓冲区比例和小批量采样策略。
原文摘要 · Abstract (English)
Double Q-learning is a classical control algorithm that mitigates the maximization bias of Q-learning. To do so, it explicitly trains two independent action-value functions and uses them to decouple action-selection and action-evaluation when computing bootstrap targets. Double DQN adapts target bootstrap decoupling to deep reinforcement learning (RL), but explicitly trains only a single action-value function and does not fully decouple its estimators. Consequently, the two estimators remain correlated, and overestimation persists. In this paper, we introduce Deep Double Q-learning (DDQL), a deep RL algorithm that explicitly trains two Q-functions through Double Q-learning. DDQL stabilizes training through a combination of techniques, including lower replay ratios, longer target network update intervals, and shared layers. Across 57 Atari 2600 games, DDQL improves aggregate performance over Double DQN, outperforming it on 47 games while further reducing overestimation. In addition, we study key design choices when adapting Double Q-learning to deep RL, including the network architecture, replay ratio, and minibatch sampling strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。