改进Q学习的目标聚合,减少不确定性下的误差放大。
Revisiting TD Target Aggregation under Uncertainty in Q-Learning
- 用动态模型的一步回溯引导动作选择,增强目标聚合的可靠性
- 在经典控制、向量环境和Atari上显著提升训练稳定性
- 无需改变框架,适合对稳定性有要求的强化学习应用
深度Q网络(DQN)通过自举式时序差分更新学习价值函数,未来回报通过下一状态动作值的贪婪最大化来近似。该方法虽有效,但对估计噪声敏感:当Q值不确定时,最大化操作会无差别地选择最大估计值,无论其可靠性如何,导致误差在自举过程中被放大。本文提出成功回溯聚合深度Q网络(SADQ),一种对Q学习的简单改进,通过正则化TD目标的形成过程来缓解此问题。SADQ利用学习到的动力学模型进行一步回溯预测,指导候选下一状态动作间的比较,在不改变底层学习框架的前提下为聚合步骤引入额外结构。由此产生的混合贝尔曼更新可抑制不可靠的最大值,同时在模型误差减小的条件下保持标准不动点。理论分析表明,SADQ以逐点方式降低自举引起的过估计。实验结果表明,相比强基线DQN变体,SADQ在经典控制任务、真实世界向量环境及Atari基准测试中均一致提升了训练稳定性。
原文摘要 · Abstract (English)
Deep Q-Networks (DQNs) learn value functions through bootstrapped temporal-difference updates, where future returns are approximated using a greedy maximization over next-state action values. While effective, this aggregation rule is inherently sensitive to estimation noise: when Q-values are uncertain, the maximization operator deterministically favors the largest estimate, regardless of its reliability, leading to amplified errors through bootstrapping. In this work, we propose the \textbf{S}uccessor Rollout \textbf{A}ggregation \textbf{D}eep \textbf{Q}-Network (SADQ), a simple modification to Q-learning that regularizes how the TD target is formed. SADQ uses one-step rollout predictions from a learned dynamics model to guide the comparison among candidate next-state actions, introducing additional structure into the aggregation step without altering the underlying learning framework. The resulting mixed Bellman update attenuates unreliable maxima while preserving the standard fixed point under diminishing model error. We provide theoretical analysis showing that SADQ reduces bootstrap-induced overestimation in a pointwise manner. Empirically, SADQ consistently improves training stability across classical control tasks, real-world vector-based environments, and Atari benchmarks when compared to strong DQN variants.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。