改进DDPG算法，让智能体在奖励稀疏时也能高效探索和学习。
ETGL-DDPG: A Deep Deterministic Policy Gradient Algorithm for Sparse Reward Continuous Control
- 引入εt-greedy搜索策略，主动发现少访问状态以增强探索。
- 设计双缓冲经验回放机制与最长n步回报，提升奖励信息利用效率。
- 适合研究稀疏奖励下的连续控制问题，尤其对强化学习初学者友好。
针对稀疏奖励场景下的深度确定性策略梯度（DDPG）算法，本文提出一种新方法。为增强探索能力，引入εt-greedy搜索机制，可生成探索性动作以访问低频状态，并在弱马尔可夫决策过程假设下证明其具有多项式样本复杂度。为更高效利用奖励触发的过渡信息，设计了双经验回放缓冲区框架GDRB，并采用最长n步回报。所提算法ETGL-DDPG融合εt-greedy、GDRB与最长n步回报三项技术。在标准基准测试中，其在所有稀疏奖励连续控制任务上均优于DDPG及现有先进方法。消融实验表明各策略独立贡献性能提升。
原文摘要 · Abstract (English)
We consider deep deterministic policy gradient (DDPG) in the context of reinforcement learning with sparse rewards. To enhance exploration, we introduce a search procedure, \emph{$ε{t}$-greedy}, which generates exploratory options for exploring less-visited states. We prove that search using $εt$-greedy has polynomial sample complexity under mild MDP assumptions. To more efficiently use the information provided by rewarded transitions, we develop a new dual experience replay buffer framework, \emph{GDRB}, and implement \emph{longest n-step returns}. The resulting algorithm, \emph{ETGL-DDPG}, integrates all three techniques: \bm{$εt$}-greedy, \textbf{G}DRB, and \textbf{L}ongest $n$-step, into DDPG. We evaluate ETGL-DDPG on standard benchmarks and demonstrate that it outperforms DDPG, as well as other state-of-the-art methods, across all tested sparse-reward continuous environments. Ablation studies further highlight how each strategy individually enhances the performance of DDPG in this setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。