用集成Q网络和梯度多样性惩罚提升离线强化学习的稳定性和精度。
SALE-Based Offline Reinforcement Learning with Ensemble Q-Networks
- 采用集成Q网络与梯度多样性惩罚，抑制分布外动作的过估计。
- 在D4RL MuJoCo上收敛更快，性能优于现有方法。
- 早期加入行为克隆，随训练进展逐步减弱，适合数据有限场景。
本文基于离线强化学习算法TD7，该算法结合状态-动作学习嵌入（SALE）与优先经验回放缓冲区（LAP）。我们提出一种无需模型的演员-评论家算法，整合集成Q网络与来自EDAC的梯度多样性惩罚。集成Q网络通过引入惩罚项引导演员网络选择分布内动作，有效应对分布外动作问题；同时，梯度多样性惩罚促使不同Q值梯度保持差异，进一步抑制分布外动作的过估计。此外，该方法保留可调节的行为克隆（BC）项，在训练初期引导演员网络趋向数据集动作，随着Q集成精度提升逐步降低其影响。这些改进协同作用，显著提升训练稳定性与精度。在D4RL MuJoCo基准上的实验表明，本方法相比现有方法具有更快的收敛速度、更高的稳定性和更强的性能表现。
原文摘要 · Abstract (English)
In this work, we build upon the offline reinforcement learning algorithm TD7, which incorporates State-Action Learned Embeddings (SALE) and a prioritized experience replay buffer (LAP). We propose a model-free actor-critic algorithm that integrates ensemble Q-networks and a gradient diversity penalty from EDAC. The ensemble Q-networks introduce penalties to guide the actor network toward in-distribution actions, effectively addressing the challenge of out-of-distribution actions. Meanwhile, the gradient diversity penalty encourages diverse Q-value gradients, further suppressing overestimation for out-of-distribution actions. Additionally, our method retains an adjustable behavior cloning (BC) term that directs the actor network toward dataset actions during early training stages, while gradually reducing its influence as the precision of the Q-ensemble improves. These enhancements work synergistically to improve the stability and precision of the training. Experimental results on the D4RL MuJoCo benchmarks demonstrate that our algorithm achieves higher convergence speed, stability, and performance compared to existing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。