针对大动作空间强化学习，提出基于优势的优化方法提升性能。
An Advantage-based Optimization Method for Reinforcement Learning in Large Action Space
- 通过优势关系建模各维度子动作，消除传统独立生成带来的偏差。
- 在HalfCheetah、Ant、Humanoid上分别获得3%、171%、84%的奖励提升。
- 适合高维动作空间任务，尤其对需精细控制的连续动作场景有效。
现实场景中的强化学习常面临高维动作空间问题，导致收敛困难、不稳定及计算复杂度高。传统基于价值的算法难以有效应对。现有方法通常在动作空间各维度独立生成子动作，但引入偏差，影响最优策略学习。本文提出基于优势的优化方法，设计了优势分支双延迟Q网络（ABQ），引入基线机制调节各维度动作值，利用子动作间的优劣势关系进行优化。该方法使每个维度的策略可独立优化。实验表明，ABQ在HalfCheetah、Ant、Humanoid环境中分别比BDQ多获得3%、171%、84%的累积奖励；与连续动作基准算法DDPG和TD3相比也表现出竞争力。
原文摘要 · Abstract (English)
Reinforcement learning tasks in real-world scenarios often involve large, high-dimensional action spaces, leading to challenges such as convergence difficulties, instability, and high computational complexity. It is widely acknowledged that traditional value-based reinforcement learning algorithms struggle to address these issues effectively. A prevalent approach involves generating independent sub-actions within each dimension of the action space. However, this method introduces bias, hindering the learning of optimal policies. In this paper, we propose an advantage-based optimization method and an algorithm named Advantage Branching Dueling Q-network (ABQ). ABQ incorporates a baseline mechanism to tune the action value of each dimension, leveraging the advantage relationship across different sub-actions. With this approach, the learned policy can be optimized for each dimension. Empirical results demonstrate that ABQ outperforms BDQ, achieving 3%, 171%, and 84% more cumulative rewards in HalfCheetah, Ant, and Humanoid environments, respectively. Furthermore, ABQ exhibits competitive performance when compared against two continuous action benchmark algorithms, DDPG and TD3.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。