arXiv:2506.05716cs.LGcs.AI2025-06

通过自适应步数与集成策略结合,缓解深度强化学习中的估值过高问题。

Ensemble Elastic DQN: A Step Dependent Ensemble Approach for Reducing Overestimation in Deep Value-Based Reinforcement Learning

  • 用Q值差异替代聚类判断,动态调整多步回报长度。
  • 短步长用集成均值,长步长用集成最小值,平衡乐观与保守。
  • 在五种MinAtar环境中四项表现最佳,适合需稳定高回报的控制任务。

深度Q网络(DQN)因基于噪声估值的最大化操作产生过估计偏差。本文提出集成弹性DQN(EEDQN),结合自适应弹性多步回报与集成目标聚合。EEDQN将早期弹性步长DQN中的聚类状态相似性检测替换为轻量级Q值差异规则,使自适应回报构建更适用于离散控制场景。算法采用时序依赖的集成聚合:单步目标使用集成均值,较长弹性回报则采用集成最小值,旨在减少极端乐观的自举估计,同时避免每一步更新过于保守。在五个MinAtar环境上对比DQN、Double DQN、Averaged DQN、MaxMin DQN和Elastic Step DQN,EEDQN在四项中取得最高最终回报,且与保守型集成基线保持竞争力。消融实验表明最优保守程度因环境而异,提示自适应步长与集成聚合存在非平凡交互关系。

原文摘要 · Abstract (English)

Deep Q-Networks (DQN) can suffer from overestimation bias because bootstrapped targets use a maximisation operation over noisy value estimates. Ensemble-based methods and multi-step methods have each been used to improve the stability and sample efficiency of value-based reinforcement learning, but their interaction remains less well understood. This paper introduces Ensemble Elastic DQN (EEDQN), a value-based reinforcement learning algorithm that combines adaptive elastic multi-step returns with ensemble-based target aggregation. EEDQN replaces the clustering-based state similarity test used in earlier Elastic Step DQN with a lightweight Q-value difference rule, making adaptive return construction simpler to apply in discrete control settings. The method then applies horizon dependent ensemble aggregation, one-step targets use the ensemble mean, while longer elastic returns use the ensemble minimum. This design aims to reduce extreme optimistic bootstrap estimates without making every update uniformly conservative. We evaluate EEDQN on five MinAtar environments against DQN, Double DQN, Averaged DQN, MaxMin DQN, and Elastic Step DQN. EEDQN achieves the highest final return in four of the five environments and remains competitive with conservative ensemble baselines. An aggregation rule ablation shows that the best degree of conservatism is environment dependent, suggesting that adaptive return length and ensemble aggregation interact in non-trivial ways.

强化学习价值函数集成方法过估计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。