揭示DQL训练不稳定的根源并提出三类稳定化方法
Understanding and Stabilizing Deep Q-Learning via Controlled Bootstrapping and Regulated Value Dynamics
- 从贝尔曼更新、贪婪选择和参数动态三方面分析不稳定性
- 发现奖励触发的自增强陷阱与参数尖峰现象
- 适合研究RL稳定性或改进DQN算法的读者
深度Q学习（DQL）在强化学习中取得了显著的实证成功，但其训练过程仍以极不稳定性著称。现有研究常将不稳定性归因于单一因素，如过估计偏差或表示学习问题，缺乏对多种不稳定性源在递归价值估计中交互作用的统一理解。本文从三个互补视角系统分析了深度Q学习中的不稳定性：贝尔曼自举中的算子级偏差、贪婪动作选择对回归噪声的估计级敏感性，以及激进数据复用下的参数动态失衡。我们识别出奖励触发的自增强陷阱与特征参数尖峰动态，并推导出受控自举、集成分位数估计和基于尖峰的参数调节等稳定化原则。在Atari-100K和Procgen上的实验表明，该方法在性能上具有竞争力，同时显著提升了训练稳定性。
原文摘要 · Abstract (English)
Deep Q-learning (DQL) has achieved remarkable empirical success in reinforcement learning, yet its training process remains notoriously unstable. Existing studies often attribute instability to isolated factors such as overestimation bias or representation learning issues, lacking a unified understanding of how different sources of instability interact during recursive value estimation. In this work, we provide a systematic analysis of instability in deep Q-learning from three complementary perspectives: operator-level bias in Bellman bootstrapping, estimator-level sensitivity of greedy action selection to regression noise, and parameter-dynamics imbalance under aggressive data reuse. We identify a reward-triggered self-reinforcing trap and characteristic parameter spike dynamics, then derive stabilization principles for controlled bootstrapping, ensemble quantile estimation, and spike-based parameter regulation. Experiments on Atari-100K and Procgen demonstrate competitive performance and improved training stability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。