根据状态探索情况动态调整动作重复，提升强化学习样本效率。
State-Novelty Guided Action Persistence in Deep Reinforcement Learning
- 基于状态新颖性动态决定动作重复次数，无需额外训练网络
- 在多种DMControl任务上显著提升采样效率，性能优于基线方法
- 可无缝集成到各类探索策略中，适合追求高效训练的研究者
深度强化学习（DRL）虽具潜力，但仍面临样本效率低的问题。为缓解探索与利用之间的权衡，已有研究采用动作持久性（即多次重复同一动作）。然而，现有方法或使用固定策略，或需学习额外的价值函数或策略来选择重复次数。本文提出一种新方法，依据当前状态空间的探索状态动态调整动作持久性，无需训练额外的函数或策略。通过平滑调度重复概率，实现更优的探索与利用平衡。该方法可无缝融入多种基础探索策略以引入时间持久性。在多个DMControl任务上的大量实验表明，所提出的基于状态新颖性的动作持久性方法显著提升了样本效率。
原文摘要 · Abstract (English)
While a powerful and promising approach, deep reinforcement learning (DRL) still suffers from sample inefficiency, which can be notably improved by resorting to more sophisticated techniques to address the exploration-exploitation dilemma. One such technique relies on action persistence (i.e., repeating an action over multiple steps). However, previous work exploiting action persistence either applies a fixed strategy or learns additional value functions (or policy) for selecting the repetition number. In this paper, we propose a novel method to dynamically adjust the action persistence based on the current exploration status of the state space. In such a way, our method does not require training of additional value functions or policy. Moreover, the use of a smooth scheduling of the repeat probability allows a more effective balance between exploration and exploitation. Furthermore, our method can be seamlessly integrated into various basic exploration strategies to incorporate temporal persistence. Finally, extensive experiments on different DMControl tasks demonstrate that our state-novelty guided action persistence method significantly improves the sample efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。