让强化学习的目标更对齐，提升训练稳定性和收敛速度。
Target-Aligned Reinforcement Learning
- 根据在线与目标网络的预测一致性筛选更新样本
- 在Atari-10上实现38.18%的性能提升
- 无需调参，仅增加不足4%运行时间
许多基于价值的深度强化学习算法依赖目标网络——在线网络的滞后副本——来稳定训练。尽管有效，这种机制带来了稳定性与信息时效性的根本权衡：目标网络更新越慢，稳定性越好，但学习信号越陈旧，阻碍收敛速度。我们提出目标对齐强化学习（TARL），一种可直接嵌入现有算法的简单改进方法，专注于那些在线网络与目标网络预测高度一致的转移样本。通过聚焦于对齐良好的目标，TARL缓解了过时目标估计的负面影响，同时保留了目标网络带来的稳定性优势。我们在多种离散和连续控制基准环境中进行了实证研究，发现TARL在不进行任何超参数调整的情况下均表现出持续改进，包括在Atari-10上实现38.18%的峰值得分提升，且运行时间增加不到4%。
原文摘要 · Abstract (English)
Many value-based deep reinforcement learning algorithms rely on target networks - lagged copies of the online network - to stabilize training. While effective, this mechanism introduces a fundamental stability-recency tradeoff: slower target updates improve stability but reduce the recency of learning signals, hindering convergence speed. We propose Target-Aligned Reinforcement Learning (TARL), a simple drop-in refinement for existing algorithms that emphasizes transitions for which the target and online network estimates are highly aligned. By focusing updates on well-aligned targets, TARL mitigates the adverse effects of stale target estimates while retaining the stabilizing benefits of target networks. We empirically demonstrate consistent improvements within discrete and continuous control algorithms across various benchmark environments without any hyperparameter tuning, including a 38.18% peak score gain on Atari-10, while incurring less than a 4% increase in wall-clock time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。