提出多步信用分配的梯度强化学习方法，提升训练速度与稳定性。
Deep Reinforcement Learning with Gradient Eligibility Traces
- 基于λ-回报扩展广义投影贝尔曼误差，支持多步信用分配
- 在MuJoCo和MinAtar上优于PPO和StreamQ，样本效率更高
- 兼顾经验回放与流式算法，适用于多种学习场景
深度强化学习中实现快速稳定的离线策略学习仍具挑战。现有方法多依赖简化高效的半梯度时序差分（TD）方法，但易导致发散。虽更严谨的梯度TD（GTD）方法具备强收敛保证，却极少用于深度强化学习。近期工作引入广义投影贝尔曼误差（$ar{ ext{PBE}}$），使GTD方法能有效处理非线性函数逼近，但仅限于一步法，信用分配慢且需大量样本。本文将$ar{ ext{PBE}}$扩展至基于λ-回报的多步信用分配，推导出三种基于梯度的方法，并提供兼容经验回放的前视形式与适用于流式算法的后视形式。实验表明，所提算法在MuJoCo和MinAtar环境中分别优于PPO和StreamQ，具有更强的样本效率与稳定性。代码已开源。
原文摘要 · Abstract (English)
Achieving fast and stable off-policy learning in deep reinforcement learning (RL) is challenging. Most existing methods rely on semi-gradient temporal-difference (TD) methods for their simplicity and efficiency, but are consequently susceptible to divergence. While more principled approaches like Gradient TD (GTD) methods have strong convergence guarantees, they have rarely been used in deep RL. Recent work introduced the generalized Projected Bellman Error ($\overline{\text{PBE}}$), enabling GTD methods to work efficiently with nonlinear function approximation. However, this work is limited to one-step methods, which are slow at credit assignment and require a large number of samples. In this paper, we extend the generalized $\overline{\text{PBE}}$ objective to support multistep credit assignment based on the $λ$-return and derive three gradient-based methods that optimize this new objective. We provide both a forward-view formulation compatible with experience replay and a backward-view formulation compatible with streaming algorithms. Finally, we evaluate the proposed algorithms and show that they outperform both PPO and StreamQ in MuJoCo and MinAtar environments, respectively. Code available at https://github.com/esraaelelimy/gtd\_algos
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。