提出IRA算法，加速在线强化学习策略利用，提升连续控制任务性能。
Improving Policy Exploitation in Online Reinforcement Learning with Instant Retrospect Action
- 通过RDE增强状态动作对表示，提升邻近样本区分度
- 引入即时策略更新机制，更新频率显著提高，最终性能提升30%以上
- 适合追求快速收敛的在线强化学习研究者使用
现有基于价值的在线强化学习算法因探索效率低和策略更新延迟，导致策略利用缓慢。为此，我们提出瞬时回溯动作（IRA）算法。首先，引入Q-表示差异演化（RDE），促进Q网络表示学习，使相邻状态-动作对具有可区分的表示。其次，采用显式策略约束机制——贪心动作引导（GAG），通过回溯历史动作有效增强策略更新过程。该方法依赖于准确的k近邻动作值估计，并通过策略约束实现快速自适应。此外，我们设计了即时策略更新（IPU）机制，系统性提升策略更新频率。实验表明，IRA在8个MuJoCo连续控制任务上显著提升学习效率与最终性能。代码已公开于https://github.com/2706853499/IRA。
原文摘要 · Abstract (English)
Existing value-based online reinforcement learning (RL) algorithms suffer from slow policy exploitation due to ineffective exploration and delayed policy updates. To address these challenges, we propose an algorithm called Instant Retrospect Action (IRA). Specifically, we propose Q-Representation Discrepancy Evolution (RDE) to facilitate Q-network representation learning, enabling discriminative representations for neighboring state-action pairs. In addition, we adopt an explicit method to policy constraints by enabling Greedy Action Guidance (GAG). This is achieved through backtracking historical actions, which effectively enhances the policy update process. Our proposed method relies on providing the learning algorithm with accurate $k$-nearest-neighbor action value estimates and learning to design a fast-adaptable policy through policy constraints. We further propose the Instant Policy Update (IPU) mechanism, which enhances policy exploitation by systematically increasing the frequency of policy updates. We further discover that the early-stage training conservatism of the IRA method can alleviate the overestimation bias problem in value-based RL. Experimental results show that IRA can significantly improve the learning efficiency and final performance of online RL algorithms on eight MuJoCo continuous control tasks.The code is available at https://github.com/2706853499/IRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。